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Abstract 

Predicting  Estimates  at  Complete  (EAC)  for  Department  of  Defense  (DoD)  space 
programs  has  proven  to  be  a  daunting  task.  Although  the  use  of  Earned  Value 
Management  (EVM)  formulations  have  been  around  for  several  decades,  research  has 
validated  the  need  to  conduct  specific  investigations  based  on  commodity  type  and 
contract  completion  percentage.  A  recent  Air  Force  Cost  Analysis  Agency  (AFCAA) 
study  improved  space  program  EAC  accuracy  using  a  Budgeted  Cost  of  Work  Perfonned 
(BCWP)  model.  This  research  was  conducted  based  exclusively  on  program  level  Work 
Breakdown  Structure  (WBS)  data.  The  DoD  requires  contractors  to  report  lower  level 
WBS  data,  and  current  guidance  supports  the  notion  that  lower  level  data  are  useful  for 
program  analysis.  This  study  assesses  the  BCWP  model  using  lower  level  WBS  data.  In 
addition,  the  second  phase  of  this  research  investigates  whether  or  not  knowledge 
concerning  lower  level  WBS  activities  can  improve  the  analytical  ability  to  predict  EAC 
cost  growth.  The  results  indicate  that  lower  level  WBS  data  does  not  improve  space 
program  EAC  accuracy  in  combination  with  the  BCWP  model.  This  research  also  finds 
that  space  programs  contain  a  great  deal  of  variability  at  lower  level  WBS  activities 
making  it  difficult  to  draw  comparisons  across  contracts. 
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FORECASTING  DOD  MID-ACQUISITION  SPACE  PROGRAM  FINAL  COSTS 


USING  WBS  LEVEL  2  AND  3  DATA 


I.  Introduction 


General  Issue 

In  today’s  ever-changing  fiscal  climate,  the  one  aspect  that  can  be  assured  is  that 
Department  of  Defense  (DoD)  major  acquisition  programs  will  need  to  be  procured  at 
cheaper  costs  and  with  more  efficient  methods.  Cost  estimating  is  one  avenue  of  the 
DoD  that  is  becoming  more  important  as  funding  limitations  continue  to  rise.  The  ability 
to  predict  final  program  cost  estimates  is  an  essential  element  in  today’s  planning  for 
tomorrow’s  future.  Space  acquisition  programs  are  no  exception  to  this  notion,  as  they 
continue  to  experience  “billions  of  dollars  in  cost  increases,  stretched  schedules,  and 
increased  technical  risks”  (Government  Accountability  Office  [GAO],  2011). 

Within  the  DoD,  declining  budgets  and  the  propensity  for  underestimation  have 
an  inverse  relationship;  that  is,  as  budgets  continue  to  decline,  the  propensity  for 
underestimating  continues  to  grow.  Programs  that  are  traditionally  funded  at  sufficient 
levels  will  see  even  more  scrutiny  of  their  annual  targets,  and  as  a  result,  competition  for 
funding  ensues.  This  idea  of  competing  for  funding  incentivizes  the  notion  of  bringing 
forth  the  lowest  revised  cost  estimate,  which  may  not  necessarily  be  the  most  reasonable 
one  (GAO,  2006).  Although  several  factors,  in  addition  to  this  incentive,  are  typically  the 
cause  of  underestimating,  poor  cost  estimating  and  prediction  techniques  continue  to  be  a 
recurring  area  of  concern. 
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Problem  Statement 


Earned  value  management  (EVM)  techniques  for  predicting  final  program  costs 
have  been  used  for  nearly  half  a  century,  yet  space  systems  acquisition  continues  to 
observe  areas  of  extreme  cost  growth  -  nearly  230  percent  in  one  recent  GAO  study 
(GAO,  2013).  The  preeminent  problem  lies  in  the  inability  to  accurately  predict  final 
space  program  costs;  this  problem  results  in  excessive  cost  growth  and  cost  overruns.  In 
the  realm  of  space  acquisition  programs,  little  research  has  been  accomplished  to 
examine  alternative  methods  to  estimating  final  program  costs  (Rusnock,  2008). 

Although  the  techniques  for  calculating  final  program  costs  have  evolved  over  time,  the 
basic  premise  remains  the  same.  This  premise  attempts  to  use  current  data  to  predict 
future  final  costs.  The  calculated  cost  is  known  as  the  program’s  Estimate  at  Complete 
(EAC).  The  EAC  value  provides  pertinent  information  that  decision  makers  require  to 
make  funding  decisions  for  an  acquisition  program.  In  situations  where  the  initial  EAC 
value  is  underestimated,  a  funding  gap  develops  between  what  was  budgeted  for  a 
program  and  what  the  program  currently  requires.  The  magnitude  of  this  gap  is 
imperative,  as  the  “affordability  of  a  single  program  can  affect  the  affordability  of  the 
entire  portfolio  of  systems”  (Keaton,  2013). 

Reasonableness,  within  the  DoD,  is  often  described  as  the  “price  that  a  prudent 
businessperson  would  pay  for  an  item  or  service  under  competitive  market  conditions” 
(Defense  Acquisition  University  [DAU],  2009).  Reasonableness,  specifically  in  terms  of 
cost  estimating,  can  be  perceived  as  how  well  an  estimate  takes  into  account  all  known 
data  at  the  time  the  estimate  is  prepared.  The  EAC  is  used  to  forecast  final  program  costs 
using  the  latest  available  perfonnance  data;  however,  an  EAC’s  reasonableness  might  be 
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valid  for  only  a  finite  period  of  time.  As  highlighted  in  one  2006  GAO  study,  it  is 
possible  for  space  programs  to  continue  for  up  to  four  years  without  an  update  to  the 
program’s  final  cost  estimates.  In  that  timeframe,  a  program  may  see  numerous  changes 
that  have  substantial  impacts  on  cost.  Some  of  these  changes  include  “changes  in 
planned  quantities,  funding  instability,  design  changes,  quality  variances  resulting  from 
rework,  manufacturing  or  engineering  changes,  changes  in  supply  chain  and  logistics 
management  and  support,  [and]  technology-related  problems”  (GAO,  2006).  As  a  result 
of  this  reasonableness  paradigm,  Air  Force  policy  requires  major  defense  acquisition 
programs  to  update  their  cost  estimates  at  least  annually  (AFI65-508,  2012). 

Due  to  the  importance  of  developing  accurate  EAC  figures,  cost  analysts  at  the 
Air  Force  Cost  Analysis  Agency  (AFCAA)  conducted  a  study  to  improve  the  accuracy  of 
EACs  for  space  programs  that  are  midway  through  their  acquisition  lifecycles.  The 
results  of  the  study  demonstrated  that  EACs  could  be  improved  by  approximately  15% 
over  the  span  representing  the  20%-70%  completion  levels  (Keaton,  2013).  This 
accuracy  improvement  is  measured  in  comparison  to  the  status  quo  EVM  “Gold  Card” 
approach.  Due  to  time  and  resource  constraints,  research  was  not  conducted  beyond  the 
Work  Breakdown  Structure  (WBS)  program  level,  and  AFCAA  detennined  that  further 
research  would  be  needed  to  detennine  the  full  extent  of  how  much  EACs  could  be 
improved  (Keaton,  2013).  The  analysis  of  lower  level  WBS  data  is  accomplished 
through  this  research. 
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Research  and  Investigative  Questions 

The  investigative  questions  aim  to  assist  in  answering  the  research  question:  can 
lower  level  WBS  data  improve  the  overall  accuracy  of  EVM  based  prediction  models  for 
mid-acquisition  space  programs?  Analysis  and  investigation  is  conducted  utilizing  both 
levels  two  and  three  WBS  space  program  data.  The  following  investigative  questions 
serve  as  the  basis  of  this  examination: 

•  What  is  the  predictive  capability  of  WBS  level  two  data  on  mid-acquisition  space 
program  EACs? 

•  What  is  the  predictive  capability  of  WBS  level  three  data  on  mid-acquisition 
space  program  EACs? 

•  Are  there  alternative  predictors  of  cost  estimates  and  growth  that  can  be  found 
through  analysis  of  WBS  two  and  three  data? 

In  answering  these  investigative  questions,  this  research  provides  a  detailed  analysis  of 
the  effectiveness  of  lower  level  WBS  data  on  forecasting  final  cost  estimates. 

Methodology 

The  methodological  approach  aims  to  explore  the  significance  of  using  WBS 
level  two  and  three  data  to  predict  actual  program  costs.  WBS  level  two  and  three  data 
are  summed  and  used  to  develop  EACs  using  several  calculation  methods.  These 
calculated  EACs  are  then  compared  to  programs’  level  one  actual  costs  to  examine  the 
accuracy  of  the  measures. 

The  investigation  is  comprised  of  two  separate  phases  of  research.  The  first  phase 
aims  to  provide  a  direct  comparison  to  the  AFCAA  study.  This  phase  analyzes  lower 
level  WBS  data,  while  using  similar  techniques  used  in  the  AFCAA  study.  The  second 
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phase  of  the  investigation  aims  to  explore  whether  or  not  commonalities  exists  at  lower 
levels  that  may  be  indicative  of  EAC  fluctuations. 

Research  Contribution 

The  predominant  focus  of  this  research  is  to  determine  the  accuracy  of  using 
lower  level  WBS  elements  to  predict  mid-acquisition  space  program  final  cost  estimates. 
The  results  of  this  research  provide  AFCAA  and  other  DoD  entities  primary  and  cross¬ 
check  analytical  tools  to  project  costs  of  space  systems  at  several  completion  percentages 
in  their  acquisition  lifecycles.  Results  of  the  study  may  not  be  generalizable  to  all 
defense  acquisition  program  types;  however,  the  results  may  provide  key  infonnation  and 
data  to  the  space  acquisition  community. 

Chapter  Summary 

This  chapter  provided  an  overview  of  the  current  fiscal  environment  within  the 
DoD,  the  importance  of  EVM  and  EAC  calculations,  and  specific  research  questions  to 
be  addressed  in  subsequent  chapters.  The  next  chapter  discusses  the  relevant  literature 
surrounding  EVM  principles  and  how  EVM  has  evolved  into  the  current  form.  Chapter 
III  describes  the  data  used,  the  sources  for  the  data,  and  the  methodologies  behind  our 
approach.  Chapter  IV  presents  the  results  of  our  analysis  and  our  research  findings. 
Lastly,  Chapter  V  presents  our  conclusions,  discusses  significant  findings  and 
implications,  and  recommends  areas  for  future  research. 
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II.  Literature  Review 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  explain  pertinent  background  infonnation  and 
review  previous  literature  concerning  Earned  Value  Management  (EVM)  and  cost 
growth.  This  chapter  first  explains  concepts  and  terminology  used  throughout  this 
research.  Following  this  explanation,  an  overview  of  previous  research  efforts  is 
provided.  This  overview  highlights  the  history  of  predicting  cost  growth,  an  investigation 
into  various  Estimate  at  Complete  (EAC)  fonnulation  methods,  and  lower  level  EVM 
data  analysis. 

Concepts 

Work  Breakdown  Structure  (WBS) 

The  WBS  is  defined  as  a  product-oriented  structural  framework  composed  of 
hardware,  software,  services,  data,  and  facilities.  It  acts  as  the  blueprint,  or  outline,  for 
the  complete  end  product  and  the  product’s  tasks.  The  WBS  begins  with  top,  or  program, 
level  data  and  segments  the  program  into  lower  level  sub-components.  At  each 
subsequent  level  of  the  WBS,  the  elements  are  more  defined.  Figure  1  displays  an 
example  of  items  seen  in  the  top  three  levels  of  a  DoD  space  program.  To  the  extent 
possible,  the  levels  of  the  WBS  should  be  mutually  exclusive  and  collectively  exhaustive. 
This  concept  necessitates  that  WBS  elements  do  not  overlap  and  that  they  capture  all 
deliverables  for  the  entire  program.  The  DoD  mandates  any  program  or  contract  to 
include  the  top  three  levels  for  reporting  purposes;  however,  complex  programs  may 
require  greater  levels  of  detail  (MIL-STD-881C,  2011).  Figure  2  displays  the  typical 
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numbered-naming  convention  used  to  define  each  WBS  level.  The  WBS  is  essential  in 
defining  the  necessary  tasks  and  building  the  framework  for  EVM. 


Level  1 


Level  2 


Level  3 


Figure  1.  DoD  Space  Program  WBS  Example 


MEL-STD-881C 

APPENDIX  F 

F.3  WORK  BREAKDOWN  STRUCTURE  LEVELS 

WBS# 

Level  1  Level  2  Level  3  Level  4 

Level  5 

1.0 

Space  System 

1.1 

SEIT/PM  and  Support  Equipment  (1  ...s)  1 

1.1.1 

Systems  Engineering 

1.1.2 

Assembly,  Integration  and  Test 

1.1.3 

Program  Management 

1.1.4 

Support  Equipment 

1.2 

Space  Vehicle  1..n  (Specify)2 

1.2.1 

SEIT/PM  and  Support  Equipment 

1.2.1. 1 

Systems  Engineering 

1.2. 1.2 

Assembly,  Integration  and  Test 

1 .2.1.3 

Program  Management 

1.2.1. 4 

Support  Equipment 

1.2.2 

Bus 

1. 2.2.1 

SEIT/PM  and  Support  Equipment 

1 .2.2.1. 1 

Systems  Engineering 

1.2.2. 1.2 

Assembly,  Integration  and  Test 

1. 2.2.1. 3 

Program  Management 

1.22.1. 4 

Support  Equipment 

1. 2.2.2 

Structures  and  Mechanisms  (SMS) 

Figure  2.  MIL-STD-881C  Space  WBS 
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Earned  Value  Management 

In  general  terms,  EVM  is  a  quantitative  project  management  technique  for 
assessing  the  performance  and  progress  of  a  program.  One  researcher,  Joseph  Lukas, 
defined  EVM  as  a  methodology  for  controlling  a  project,  while  relying  on  the  WBS  work 
performance  and  an  integrated  schedule  and  budget  (Lukas,  2008).  Lukas  frames  his 
argument  on  the  concept  of  differentiating  among  earned  value  analysis,  EVM,  and  an 
earned  value  management  system.  For  the  purpose  of  this  research,  EVM  is  used  as  an 
all-encompassing  tenn  to  describe  the  overall  subject  matter. 

EVM  data  are  the  product  of  various  fonnulas  and  calculations  linked  to  three  key 
cost  elements:  Budgeted  Cost  of  Work  Performed  (BCWP),  Budgeted  Cost  of  Work 
Scheduled  (BCWS),  and  Actual  Cost  of  Work  Performed  (ACWP).  The  BCWP 
represents  the  “value  of  the  work  actually  completed”;  the  BCWS  represents  the  “portion 
of  the  approved  cost  estimate  planned  to  be  spent  on  the  project  activity  in  a  given  time 
period”;  and  the  ACWP  represents  the  “total  of  costs  incurred  in  accomplishing  work  on 
the  activity”  (Godbole,  2004).  These  three  elements  function  as  the  basic  foundation  for 
EVM;  remaining  EVM  measurements  are  computed  using  a  combination  of  these  three 
elements.  A  detailed  description  of  common  EVM  tenns  and  their  calculations  is  found 
in  Table  1,  which  is  a  reproduction  of  information  found  in  the  Defense  Acquisition 
Guidebook  (DAU,  2013). 
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Table  1.  EVM  Terms 


EVM  Term 

Meaning 

Formula 

BCWP 

"Earned  value";  how  much  budgeted  cost  the 
program  has  gained 

Sum  of  the  budgeted  cost  of  all  completed 
work  packages 

BCWS 

"Planned  value";  how  much  budgeted  value  the 
program  should  have  gained 

Sum  of  the  budgeted  cost  of  all  work 
packages  scheduled 

ACWP 

"Actual  cost";  how  much  cost  has  been  incurred 
for  the  completed  work  packages 

Sum  of  actual  costs  of  all  completed  work 
packages 

Cost  Variance  (CV) 

Difference  between  planned  and  actual  cost  to  date 

BCWP  -  ACWP 

Schedule  Variance  (SV) 

Difference  (expressedin  dollars)  between  planned 
and  actual  schedule  to  date 

BCWP  -  BCWS 

Cost  Performance  Index 
(CPI) 

Cost  efficiency  of  the  program 

BCWP  /  ACWP 

Schedule  Performance 
Index  (SPI) 

Schedule  efficiency  of  the  program 

BCWP  /  BCWS 

Budget  at  Complete 
(BAC) 

Planned  total  cost  of  the  program 

Sum  of  all  BCWS  of  program 

Budgeted  Cost  of  Work 
Remaining  (BCWR) 

Budgeted  cost  of  uncompleted  work  packages  to 
reach  program’s  completion 

BAC  -  BCWP 

EAC 

Forecasted/estimated  total  cost  of  program 

Various  formulas 

To  Complete  Performance 
Index  (TCPI) 

Projected  CPI  for  the  remainder  of  the  project  to 
meet  the  BAC 

Various  formulas 

Variance  at  Complete 
(VAC) 

Estimated  cost  variance  at  completion  of  program 

BAC  -  EAC 

Percent  Complete 

Percentage  of  the  entire  program  that  is  complete 

BCWPcum  /  BAC 

Total  Allocated  Budget 
(TAB) 

Total  of  all  contract’s  work  budgets 

Sum  of  all  budgets 

Contract  Base  Budget 
(CBB) 

Total  budget  allotted  to  the  contractor 

Sum  of  the  negotiated  contract  cost  and 
authorized  undefined  work 

Management  Reserve 
(MR) 

Amount  of  the  budget  allotted  for  unknown  costs 
or  risk  management 

Determined  at  start  of  contract 

Over  the  course  of  the  past  50  years,  the  analysis  of  EVM  data  has  proven  to  be 
an  adequate  technique  for  controlling  cost,  schedule,  and  perfonnance  for  government 
contracts  (Defense  Contract  Management  Agency  [DCMA],  2006).  One  area  of 
controlling  costs,  involves  predicting  program  EACs.  Although  it  is  not  explicitly  stated, 
Table  1  implicitly  denotes  that  EACs  can  be  calculated  using  various  methodologies  and 
techniques.  The  primary  source  for  obtaining  program  specific  EVM  data  from 
contractors  is  the  contractor  perfonnance  report  (CPR).  The  CPR  serves  as  the  unbiased 
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reporting  method  for  delivering  program  status  updates  to  the  government  and  program 
analysts. 

Contractor  Performance  Report 

The  CPR  is  a  management  document  that  provides  summary-level  EVM  data  to 
DoD  system  managers.  Managers  are  then  able  to  compare  cost  and  schedule 
performance  data  with  technical  performance  measures,  identify  both  actual  and  potential 
problem  areas,  and  provide  valid,  timely  program  updates  to  senior  leaders.  CPRs  come 
in  seven  different  fonnats,  as  seen  in  Table  2  (OUSD:  AT&L,  2005).  This  research 

Table  2.  CPR  Formats 


Title 

Requirement 

Description 

Format  1  -  WBS 

Mandatory 

Provides  data  to  measure  cost  and  schedule 
perfonnance  by  product-oriented  WBS 
elements 

Format  2  - 
Organizational 
Categories 

Optional 

Provides  the  same  data  as  Fonnat  1,  but  by  the 
contractor's  organization 

Format  3  -  Baseline 

Optional 

Provides  the  budget  baseline  plan  against 
which  perfonnance  is  measured 

Format  4  -  Staffing 

Optional 

Provides  staffing  forecasts  for  correlation  with 
the  budget  plan  and  cost  estimates 

Format  5  -  Problem 
Areas 

Mandatory 

Provides  narrative  information  used  to  explain 
significant  cost  and  schedule  variances  and 
other  identified  contract  problems  and  topics 

Format  6  -  Integrated 
Master  Schedule 

Optional 

Provides  details  as  to  how  and  when  the 
integrated  master  plan  (IMP)  is  accomplished 

Fonnat  7  -  Electronic 
History  and  Forecast  File 

Mandatory 

Provides  time  phased  cost  data  to  include  both 
historical  and  forecasted  information 

focuses  on  Format  1,  the  WBS  approach.  An  example  of  the  Fonnat  1  CPR  can  be  seen 
in  Appendix  A.  The  WBS  CPR  is  required  at  least  monthly,  but  can  be  required  more 
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frequently  in  certain  circumstances.  This  CPR  provides  the  necessary  data  to  conduct 
earned  value  analyses  at  each  of  the  first  three  levels  of  the  WBS.  Research  shows  that 
these  analyses  are  vital  in  continually  improving  EAC  estimation  and  the  identification  of 
overall  program  growth  (Rosado,  2011). 

Relevant  Research 

In  the  research  study,  “Predicting  Cost  and  Schedule  Growth  for  Military  and 
Civil  Space  Systems,”  Rusnock  attempts  to  develop  a  set  of  models  that  can  be  used  to 
predict  space  system  cost  and  schedule  growth.  Rusnock  separately  analyzes  both  DoD 
and  National  Aeronautics  and  Space  Administration  (NASA)  space  programs  to 
detennine  which  characteristics  provide  the  best  indication  as  to  whether  or  not,  and  to 
what  extent,  cost  and  schedule  growth  will  occur.  Rusnock  finds  that  communications 
missions,  ground  equipment,  firm-fixed  price  contracts,  and  increased  program  manager 
tenure  are  all  found  to  be  predictive  of  lower  cost  growth  for  DoD  space  programs. 
Rusnock  provides  various  cost  growth  regression  equations  and  possible  explanations  for 
the  existence  of  the  relationships  (Rusnock,  2008).  Rusnock’s  research  effort  provides 
models  and  characteristics  of  programs  that  pose  a  risk  for  cost  growth;  however,  with 
this  infonnation  alone,  analysts  can  not  make  strategic  decisions  to  mitigate  these  risks. 
For  example,  analysts  do  not  generally  have  control  over  the  commodity  type,  contract 
type,  or  length  of  PM  tenure;  therefore,  analysts  would  be  able  to  use  Rusnock’s  findings 
only  to  assess  whether  or  not  their  program  contains  attributes  that  are  consistent  with 
lower  or  higher  levels  of  cost  growth.  Although  useful,  Rusnock’s  research  does  not 
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provide  analysts  with  tools  that  they  can  utilize  to  convey  final  EACs  to  upper 
management. 

EAC  fonnulas  and  calculation  approaches  are  abundant  within  the  acquisitions 
community.  It  is  often  difficult  for  PMs  and  analysts  to  detennine  which  methods  are 
appropriate  for  their  particular  program.  In  a  1995  review  by  Christensen  et  ah,  this 
difficulty  level  was  further  reinforced.  The  paper  reviews  25  studies  on  EAC  formulas 
and  models  to  detennine  which  method  is  the  most  accurate.  The  results  indicate  that  the 
accuracy  levels  of  the  various  approaches  are  dependent  upon  several  factors;  therefore, 
no  one  method  was  deemed  to  be  the  most  accurate  in  all  settings.  Index -based  fonnulas’ 
accuracy  was  shown  to  be  dependent  on  both  the  system  type  and  the  contract  completion 
stage  (Christensen  et  ah,  1995).  These  results  enforce  the  need  for  independent  research 
that  compares  various  EAC  calculation  methods  across  specific  system  types  at  different 
completion  percentages.  As  noted  in  the  preceding  AFCAA  research  study,  current  space 
system  cost  estimation  techniques  using  EVM  methods  are  particularly  sparse  (Keaton, 
2013). 

The  research  study  by  AFCAA  analysts  focuses  on  predicting  EACs  for  space 
programs  using  program  level  data;  however,  the  author  also  suggests  that  subsequent 
level  data  could  provide  useful  analysis  as  well  (Keaton,  2013).  The  DCMA  EVM 
System  guidance  supports  this  suggestion  when  it  states  that  detailed  analysis  on 
contractor  EACs  should  be  performed  at  the  lowest  WBS  level  available  (DCMA,  2012). 
In  201 1,  Rosado  conducted  the  first  documented  research  study  aimed  at  detennining  the 
predictive  capability  of  lower  level  EVM  data  using  regression  based  techniques.  The 
study  focuses  solely  on  Research,  Development,  Test  and  Evaluation  (RDT&E) 
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contracts.  Rosado  discovers  that  there  are  little  to  no  WBS  lower  level  commanlities 
between  contracts;  this  holds  true  even  within  the  same  service,  commodity  type,  and 
contractor.  With  the  one  commonality  he  is  able  to  find,  level  three  Development  Test 
and  Evaluation,  the  data  prove  to  be  significant  drivers  for  program  EAC  growth; 
however,  these  findings  are  not  generalizable  to  all  DoD  acquisition  contracts.  Rosado 
also  notes  that  the  results  are  not  dependent  upon  service,  contractor,  or  acquisition 
category.  Lastly,  Rosado  expresses  that  future  research  should  provide  analysis  based  on 
specific  program  characteristics  such  as  commodity  type.  (Rosado,  2011). 

Although  the  accuracy  of  calculated  EACs  can  vary  greatly,  it  is  important  to 
recognize  the  propensity  for  PMs  and  contractors  to  select  the  most  optimistic  estimates 
to  bring  forward.  Christensen  articulates  this  tendency  in  his  article  “Project  Advocacy 
and  the  Estimate  at  Completion  Problem.”  Christensen  uses  the  A-12  program  to 
demonstrate  how  optimism,  by  both  contractors  and  government  project  managers  (PM), 
can  be  dangerous,  which  ultimately  led  to  the  A-12  program’s  cancellation.  Christensen 
believes  that  contractors  and  PMs  knowingly  provide  optimistic  EAC  figures  in  an  effort 
to  prolong  programs  by  securing  funding.  The  results  of  Christensen’s  research  effort 
show  that  the  cumulative  Cost  Performance  Index  (CPI),  which  had  been  commonly  used 
as  a  ceiling  in  numerous  estimates,  was  in  fact  a  floor  to  the  average  final  costs.  The 
research  also  showed  that  there  was  a  significant  correlation  between  the  reported  EACs 
and  the  estimates  calculated  using  cumulative  CPI.  Other  common  index -based  estimates 
were  proven  to  be  more  accurate;  however,  they  were  less  correlated  to  the  reported 
EACs.  These  results  suggested  that  although  more  accurate  methods  existed,  PMs  and 
contractors  knowingly  chose  the  lowest  estimate  as  a  basis  for  their  reported  figures 
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(Christensen,  1996).  This  inclination  to  report  the  estimate  that  presents  the  most 
favorable  outcome  represents  an  inherent  limitation  to  any  EAC  prediction  model 
research  study.  One  theory  suggests  that  “perhaps  the  future  of  this  discipline  [cost 
estimation]  will  focus  on  methods  of  controlling  program  cost,  rather  than  the  futile 
attempt  to  predict  it”  (Keller  et  ah,  2014). 

This  theory  on  controlling  program  costs  is  the  concluding  thought  of  a  recent 
study  by  Keller  et  ah,  “What  is  Wrong  with  Space  System  Cost  Models?  A  Survey  and 
Assessment  of  Cost  Estimating  Approaches,”  which  initially  aimed  to  gather  a  deeper 
understanding  of  the  causes  cost  growth  in  space  programs.  Keller  et  al.  find  that  current 
methods  and  approaches  to  cost  estimating  space  programs  are  inadequate  due  to  their 
inability  to  predict  the  future,  lack  of  insight,  and  the  idea  of  processes  replacing  good 
judgment  in  decision  making.  The  authors  first  state  that  parametric  models  lack  the 
ability  to  extrapolate  beyond  the  programs  of  the  past  which  they  are  founded  upon.  Next 
the  authors  state  that  prediction  models  and  techniques  do  not  possess  the  ability  to 
accurately  define  cause  and  effect  relationships  in  cost  growth.  Without  this  ability,  it 
becomes  rather  difficult  to  improve  upon  final  cost  estimation  modeling.  Lastly,  the 
article  explains  that  analysts  are  too  overwhelmed  by  the  possibility  of  program  failure. 
This  fear  drives  analysts  to  accept  the  mindset  that  insight  and  accuracy  are  “less 
important  than  providing  a  number  that  will  be  approved  by  senior  management  and  the 
Office  of  Management  and  Budget”  (Keller  et  al.,  2014). 
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Summary 

In  summary,  there  is  a  long  history  of  cost  growth  models  and  EAC  formulation 
practices.  In  order  to  provide  maximum  value,  new  research  must  be  founded  on 
infonnation  that  is  readily  available  to  program  analysts,  and  the  findings  should  be 
quantitative  in  nature  so  that  analysts  can  provide  useful  infonnation  to  their  leadership. 
Through  his  research,  Christensen  established  the  need  to  conduct  EAC  analysis  on 
specific  system  types  and  at  various  completion  percentages.  Lower  level  EVM  data  has 
shown  to  be  rather  inconsistent  when  making  comparisons  of  various  system  types  and 
across  services;  however,  deeper  analyses  can  uncover  elements  that  may  be  indicative  of 
program  cost  growth.  Taking  all  this  into  consideration,  the  inclination  to  report  the 
estimate  that  presents  the  most  favorable  outcome  represents  an  inherent  limitation  to  any 
EAC  research  study.  This  mentality  has  even  led  some  to  believe  that  we  should  stop 
trying  to  predict  space  program  costs  altogether.  The  next  chapter  discusses  the  data 
sample  and  methodological  approaches  taken  to  address  the  issue  of  developing  better 
space  program  estimates. 
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III.  Data  Collection  and  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  provide  background  infonnation  on  the  data 
collection  and  research  methodology  for  the  analyses  in  this  thesis.  This  chapter  first 
provides  a  detailed  explanation  of  the  sample  data  collected.  It  then  explains  the 
methodological  approach  taken  and  formulas  used  to  conduct  data  analysis.  The  primary 
tools  utilized  for  this  research  are  Microsoft  Excel  and  the  Defense  Cost  and  Resource 
Center  (DC ARC)  CPR  Viewer. 

Data  Source 

The  dataset  used  in  this  research  is  comprised  of  program  data  drawn  from  the 
DCARC  EVM  central  repository.  DCARC  was  established  in  1998  and  is  the  largest 
source  of  DoD  cost  data  available  to  the  cost  community.  DCARC’s  primary  mission  is 
to  “collect  current  and  historical  Major  Defense  Acquisition  Program  (MDAP)  and  Major 
Automated  Information  System  (MAIS)  cost  and  software  resource  data  in  a  joint  service 
environment”  (DCARC,  2007).  These  program  data  were  assembled  from  monthly 
CPRs,  contract  history  files,  data  reports,  Integrated  Program  Management  Reports,  and 
other  miscellaneous  submissions  generated  from  program  offices  and  contractors. 

Due  to  the  necessity  of  having  data  at  WBS  levels  two  and  three,  many  of  the 
EVM  data  had  to  be  manually  transcribed  into  Excel;  however,  the  primary  method  of 
gathering  level  two  and  three  data  consisted  of  downloading  the  latest  history  file  and 
opening  the  file  using  the  DCARC  CPR  File  Viewer.  The  “export  to  Excel”  function  was 
utilized  to  put  the  data  in  a  useable,  easily  modifiable  fonnat.  The  selected  data  are 
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broken  into  two  distinct  samples.  The  smaller  sample,  dataset  one,  consists  of  the  six 
space  acquisition  contracts  used  in  the  previous  AFCAA  study  (Keaton,  2013).  The 
larger  sample,  dataset  two,  contains  all  space  program  contracts  found  within  the 
DCARC  repository. 

Dataset  one  is  linked  to  the  preceding  research  study  in  which  Keaton  selected 
programs  that  were  representative  of  the  field  and  contained  sufficient  data  to  conduct 
their  research.  Space  acquisitions,  as  a  whole,  is  a  vast  field  of  study,  and  there  are  many 
different  focuses  within  its  realm;  some  of  the  more  common  areas  include  space 
vehicles,  ground  systems,  satellites,  ground  control  centers,  and  shuttles.  For  this  reason, 
the  previous  research  study  sample  was  used  to  focus  the  study  on  areas  deemed  most 
important  to  field  experts.  Furthermore,  one  of  the  main  goals  of  this  research  is  to 
determine  the  predictive  capability  of  level  two  and  three  data.  Taking  this  concept  into 
account,  the  use  of  the  six  contracts  used  in  the  AFCAA  study  would  allow  for  a 
conclusive  assessment  of  the  predictability  of  level  two  and  three  data  in  comparison  to 
the  AFCAA  study  which  used  level  1  data.  Table  3  displays  the  contracts  that  are 
considered  for  dataset  one. 

Table  3.  Dataset  One  Contracts 

Program  Name 

AEHF  -  Advanced  Extremely  High  Frequency  (AEHF)  Satellite  Program 

GPS  El  -  Global  Positioning  Satellite  m 

SBIRS  HIGH  -  Space-Based  Infrared  System  Program,  High 

WGS  -  Wideband  Global  SATCOM  Program 

MUOS  -  Mobile  User  Objective  System 

FAB-T  -  Family  of  Beyond  Line-of-Sight  Terminals 
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Unfortunately,  one  of  the  six  contracts,  Space-Based  Infrared  System  (SBIRS), 
could  not  be  assessed  at  the  level  two  and  three  WBS  levels  due  to  inaccessibility  of  the 
data.  Attempts  were  made  to  acquire  the  data  from  AFCAA,  but  the  data  were  not 
available  in  a  useable  format.  In  addition  to  the  inaccessibility  of  the  SBIRS  data,  the 
program  exhibited  traits  that  would  have  made  it  difficult  to  assess.  Most  notably,  the 
contract  completion  percentages  experienced  a  “roller-coaster”  effect  when  graphed  due 
to  the  number  of  Over  Target  Baselines  (OTB).  This  is  clearly  seen  in  Figure  3. 


0  50  100  150  200 

Month  Count 

Figure  3.  SBIRS  OTB  Effects 

Dataset  two  initially  encompasses  all  space  program  contracts  found  within 
DC  ARC.  This  initial  assessment  results  in  41  total  space  contracts.  9  contracts  are 
removed  due  to  their  absence  of  CPR  data  within  DC  ARC.  15  contracts  are  removed  due 
to  not  being  at  least  85%  complete  in  terms  of  BCWP  and  BAC.  An  85%  completion 
limitation  is  placed  on  the  data  set  in  order  to  ensure  that  the  contract  is  somewhat 
nearing  completion.  Although  previous  research  suggests  that  the  92.5%  completion 
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point  is  a  better  representation  of  a  contract’s  final  costs  (Tracy  and  White,  2011),  the 
85%  point  is  used  in  order  to  include  as  many  contracts  as  possible.  Imposing  this  lower 
rate  limitation,  results  in  three  additional  contracts  being  retained  in  the  dataset;  two  of 
which  were  used  in  the  preceding  AFCAA  study.  Lastly,  8  contracts  are  removed  due 
containing  missing,  insufficient,  or  inaccurate  data  at  WBS  levels  two  and  three.  Of  the 
4 1  total  space  contracts,  only  9  contracts  meet  the  final  specifications  for  this  research. 
Table  4  illustrates  this  gradual  decline  in  the  number  of  contracts  used  in  dataset  two. 


Table  4.  Dataset  Two  Contracts 


Proaram  Name 

Contracts 

Considered 

w/  CPR 
Data 

>85% 

Complete 

Met  Criteria 

*AEHF  -  Advanced  Extremely  Hiqh  Frequency  (AEHF)  Satellite  Proqram 

2 

2 

1 

1 

EELV  -  Evolved  Expendable  Launch  Vehicle 

2 

2 

2 

1 

EPS  -  Enhanced  Polar  System 

2 

1 

0 

- 

*GPS  in  -  Global  Positioninq  Satellite  HI 

2 

2 

1 

1 

GPS  OCX  -  Global  Positioning  Satellite  Next  Generation  Control  Segment 

3 

3 

2 

1 

JTN  -  Joint  Tactical  Networks 

5 

2 

2 

0 

MGUE  -  Military  GPS  User  Equipment 

3 

3 

1 

0 

NAVSTAR  GPS  -  Global  Positioning  System 

4 

4 

4 

1 

NPOESS  -  National  Polar-Orbiting  Operational  Environmental  Satellite 
System 

1 

1 

0 

- 

SBIRS  HIGH  -  Space-Based  Infrared  System  Program,  High  Component 

7 

5 

1 

0 

*WGS  -  Wideband  Global  SATCOM  Proqram 

2 

2 

2 

2 

*MU0S  -  Mobile  User  Objective  System 

1 

1 

1 

1 

*FAB-T  -  Family  of  Beyond  Line-of-Siqht  Terminals 

6 

3 

1 

1 

Space  Fence 

1 

1 

0 

- 

Total 

41 

32 

18 

9 

The  five  programs  marked  with  an  asterisk  are  utilized  in  both  finalized  datasets. 
The  final  9  contracts  in  dataset  two  stem  from  8  of  the  total  14  different  space  programs 
in  DCARC.  This  is  significant  since  a  large  percentage  of  contracts  did  not  meet  the  final 
criteria,  yet  over  half  of  all  the  space  programs  in  DCARC  are  still  represented  in  this 
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study.  The  contract  start  dates  for  both  datasets  span  over  a  period  of  approximately  14 
years.  The  quantity  by  time  period  is  captured  in  Table  5. 

Table  5.  Number  of  Contracts  by  Contract  Start  Date 


Contract  Start  Date 

Number  of 
Contracts 

1  Jan  2000-31  Dec  2004 

3 

1  Jan  2005-31  Dec  2009 

4 

1  Jan  2010-31  Dec  2013 

2 

Total 

9 

Descriptive  statistics  regarding  the  various  amounts  of  contract  completion 
percentages  are  found  in  Table  6.  Although  the  85%  completion  point  was  used  as  the 
cutoff,  the  mean  final  reported  EAC  point  was  96%  for  the  9  contracts.  The  completion 
percentages  for  each  contract  can  be  seen  by  contract  in  Figure  4.  These  completion 
percentages  are  representative  of  the  available  level  one  data.  On  occasion,  subsequent 
level  data  is  not  available  at  this  same  completion  level;  however,  this  results  in  a 
difference  of  only  a  few  percentage  points.  For  example,  the  AEHF  contract  has  level 
one  data  available  up  until  the  99%  completion  point;  however,  the  level  two  and  three 
data  is  available  up  until  only  the  95%  completion  point.  This  difference  is  caused  by  a 
lag  in  history  file  reporting  within  DC  ARC. 

Additionally,  there  are  two  contracts  that  contain  data  issues  worth  noting. 
Although  most  of  the  data  used  in  this  research  is  taken  from  the  primary  source, 
DCARC,  one  contract  necessitated  the  use  of  a  secondary  source.  The  FAB-T  contract 
began  in  September  2002,  yet  the  data  available  in  DCARC  does  not  begin  until  October 
2006.  Hence,  the  data  for  this  contract  was  supplied  by  AFCAA.  Secondly,  the 
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NAVSTAR  contract  encounters  a  unique  occurrence  late  into  its  duration.  At  the  92% 
completion  point,  the  nomenclature  of  the  level  two  and  three  WBS  activities  drastically 
changes  beyond  the  point  of  reconciliation.  Many  of  the  previous  WBS  activities  begin 
with  a  new  EAC  figure  of  zero  and  remain  there  for  several  reporting  periods.  For  these 
reasons,  the  portion  of  analysis  using  lower  level  data  covers  the  period  up  until  this 
point. 


Table  6.  Contract  Coverage  Statistics 


Minimum  EAC  Coverage 

87% 

Maximum  EAC  Coverage 

100% 

Median  EAC  Coverage 

99% 

Mean  EAC  Coverage 

96% 

Standard  Deviation 

5% 

Figure  4.  Contract  Completion  Percentages 
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Methodology 

The  methodological  approach  taken  for  this  thesis  consists  of  two  phases.  The 
first  phase  makes  use  of  dataset  one  and  is  primarily  influenced  by  the  preceding  study. 
The  goal  of  this  approach  is  to  recreate  similar  procedures  to  those  used  in  the  AFCAA 
study  in  an  effort  to  make  a  direct  comparison  to  the  results  derived  from  utilizing  lower 
level  EVM  data.  By  conducting  the  analysis  in  this  manner,  variations  in  the  results  of 
the  two  studies  can  be  directly  attributed  to  the  use  of  level  two  and  three  data.  This  is 
important  so  that  disparities  are  not  credited  to  differences  in  the  procedural  or  technical 
approaches.  The  second  phase  of  the  methodology  aims  to  explore  the  extent  of 
underlining  relationships  found  within  lower  level  EVM  data.  Phase  two  provides 
detailed  analysis  that  searches  beyond  the  EVM  summary  level  two  and  three  totals.  In 
this  phase,  the  research  involves  looking  into  EAC  cost  drivers,  finding  commonalities 
between  programs  and  contracts,  and  seeing  how  changes  in  those  drivers  affect  the  EAC 
estimates. 

Phase  One 

Phase  one  of  the  methodological  approach  consists  of  steps  that  are  comparable  to 
those  used  in  the  preceding  study.  First,  WBS  level  one,  two,  and  three  EVM  data  are 
gathered  for  each  contract  and  placed  in  an  Excel  worksheet.  These  data  consist  of  the 
contract  start  date,  estimated  completion  date,  BAC,  EAC,  BCWS,  BCWP,  and  the 
ACWP.  Next,  the  completion  points  are  determined  for  each  contract.  Completion 
points,  in  10%  increments,  are  identified  from  the  20%  point  to  the  70%  contract 
completion  point.  Following  this  identification  process,  the  cumulative  Cost 
Performance  Index  (CPI)  and  Schedule  Perfonnance  Index  (SPI)  performance  measures 
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are  calculated  for  each  contract  at  every  monthly  reporting  period.  In  fonnulating  these 
measures,  a  perfonnance  baseline  EAC  can  be  established.  This  baseline  is  considered  as 
the  standard,  or  traditional,  approach  with  which  most  members  of  the  cost  community 
are  familiar  (Keaton,  2013).  The  formula  for  this  baseline  is  found  on  the  Defense 
Acquisition  University  (DAU)  EVM  “Gold  Card”  and  is  known  as  the  EAC  Composite: 


EAC, 


BAC 


Composite 


c  P I  Cum  *  SP  f 


Cum 


BAC 

~scT 


(1) 


The  use  of  the  EAC  Composite  as  the  baseline  is  a  critical  aspect  of  this  research. 
It  is  often  thought  that  this  EAC  represents  the  cost  ceiling  in  most  programs  which  are 
traditionally  over  cost  and  behind  schedule.  As  stated  in  the  Defense  Management 
Contractor  Agency  (DCMA)  EVM  System  Program  Analysis  Pamphlet,  “the  EAC 
Composite  fonnula  provides  an  upper  bound,  or  the  most  pessimistic  IEAC  [Independent 
EAC]”  (DCMA,  2012).  Hence,  if  a  program  is  severely  over  budget  and  exceeds 
estimated  initial  costs,  the  EAC  Composite  should  represent  the  most  accurate  figure,  in 
terms  of  proximity,  of  the  traditional  EVM  based  estimates. 

The  next  segment  of  phase  one  consists  of  calculating  EACs  using  the  BCWP 
model.  The  BCWP  model  utilizes  the  relationship  between  remaining  contract  months, 
the  BCWP  bum  rate,  and  BCWP  to  date  to  estimate  final  costs.  Final  cost  predictions  for 
contracts  are  made  from  the  20%  -  70%  incremental  completion  points.  This  approach 
may  vary  from  other  EVM  approaches  because  it  “focuses  exclusively  on  the  rate  at 
which  work  is  accomplished  rather  than  cost  efficiency”  (Keaton,  2013).  In  other  words, 
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this  approach  assumes  linearity.  The  relationship  among  the  variables  is  displayed  in  the 
following  equation: 


EACgcyyp  —  (MonthEst  compietion  M OnthEurrent )  *  BCW Psum  Rate  "F  BC WPj0  Date  (2) 


In  the  AFCAA  study,  the  remaining  number  of  months  is  derived  as  a  function  of 
the  total  number  of  estimated  months  and  the  earned  schedule  up  until  the  respective 
completion  point.  The  total  number  of  estimated  months,  or  program  duration,  for  each 
contract  is  calculated  by  taking  the  difference  between  the  most  recent  estimated 
completion  date  and  the  contract  start  date.  The  current  month  portion  of  the  BCWP 
model  is  calculated  based  on  the  previous  period’s  earned  schedule,  plus  the  earned 
schedule  for  the  current  period: 


Earned  Schedule  =  ESPrev  + 


( BCWPCurrent  BCWPPrev ) 
(BCWSCurrent  —  BCWSPrev ) 


(3) 


In  addition  to  this  approach  of  evaluating  the  remaining  number  of  months,  the  actual 
number  of  estimated  remaining  months  is  also  assessed  in  the  BCWP  model.  For 
example,  at  the  70%  completion  point  for  the  FAB-T  contract,  the  estimated  program 
duration  is  1 1 1  months.  At  this  same  point,  the  earned  schedule  is  calculated  at  26 
months,  while  the  actual  month  count  is  69  months.  Using  the  earned  schedule  approach 
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would  result  in  an  estimated  number  of  remaining  months  of  85.  Using  the  actual 
figures,  results  in  an  estimated  number  of  remaining  months  of  42. 

The  BCWP  model  utilizes  ordinary  least  squares  regression  to  calculate  the 
BCWP  burn  rate.  This  bum  rate  is  simply  the  coefficient  from  a  prediction  model 
generated  from  the  elapse  number  of  months  and  the  cumulative  BCWP  figures.  At  each 
of  the  measured  completion  points,  20%-70%,  the  elapsed  months  are  used  as  the 
explanatory  variable  to  predict  the  response,  the  cumulative  BCWP.  For  example,  if 
Contract  A  reaches  the  20%  completion  point  at  13  months,  a  prediction  model  is  derived 
using  each  of  the  elapsed  months,  1  through  13,  as  predictors  for  their  respective 
cumulative  BCWP.  The  derived  coefficient  for  the  elapsed  months  is  used  in  the  BCWP 
model  as  the  bum  rate.  In  Figure  5,  this  burn  rate  would  be  approximately  $20,629,032. 

20%  SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.997726669 

R Square 

0.995458506 

Adjusted  R  Square 

0.995045643 

Standard  Error 

5667688.673 

Observations 

13 

ANOVA 


df 

SS 

MS 

F 

Significance  F 

Regression 

1 

7.74514E+16 

7.74514E+16 

2411.110414 

3.06772E-14 

Residual 

11 

3.5335E+14 

3.21227E+13 

Total 

12 

7.78047E+16 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Intercept 

-10545492.3 

3334575.593 

-3.1624691 

0.009039083 

-17884843.66 

X  Variable  1 

20629031.63 

420117.0356 

49.10305911 

3.06772E-14 

19704360.27 

Figure  5.  BCWP  Regression  Coefficient  Example 


In  addition  to  this  approach  of  evaluating  the  burn  rate,  the  monthly  average  burn  rate  is 
also  assessed  in  the  BCWP  model.  For  example,  if  the  cumulative  BCWP  for  the  same 
Contract  A  is  $266,700,245  at  the  20%  completion  point,  then  the  burn  rate  would  be 
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$20,5 15,404.  This  is  derived  from  simply  dividing  the  cumulative  BCWP  by  the  number 
of  elapsed  months. 

As  a  result  of  the  variations  to  the  original  BCWP  bum  rate  formula,  Equation  2, 
three  additional  models  are  tested.  Model  1  is  the  fonnula  as  it  was  utilized  in  the 
previous  research  study.  The  BCWP  bum  rate  is  calculated  using  the  regression  based 
approach,  and  the  remaining  number  of  months  is  calculated  based  on  the  estimated 
number  of  total  months  minus  the  earned  schedule  up  until  that  point  in  time.  Model  2  is 
similar  to  Model  1 ;  however.  Model  2  uses  the  actual  schedule  in  place  of  the  earned 
schedule  to  calculate  the  remaining  number  of  months.  Model  3  uses  the  monthly 
average  BCWP  burn  rate  in  place  of  the  regression  based  burn  rate.  Model  4  uses  the 
monthly  average  BCWP  burn  rate  and  the  actual  schedule.  Table  7  displays  the  details  of 
all  four  models. 


Table  7.  BCWP  Model  Variations 


Model 

BCWP  Burn  Rate 

Remaining  Months 

Model  1 

Regression 

Earned  Schedule 

Model  2 

Regression 

Actual  Schedule 

Model  3 

Average 

Earned  Schedule 

Model  4 

Average 

Actual  Schedule 

After  calculating  the  various  predicted  EAC  figures,  the  accuracy  percentages  are 
measured  based  on  the  numerical  proximity  to  the  final  EAC.  The  accuracy  of  the 
calculated  figures  is  measured  using  an  absolute  percent  error  measurement.  This 
measurement  formulates  the  percentage  of  error  by  dividing  the  difference  between  the 
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final  and  predicted  EAC  values  by  the  last  reported  EAC.  This  number  is  then  multiplied 
by  100  to  develop  a  percentage  of  error.  As  an  example,  a  zero  percent  error  rate  would 
mean  that  the  model  predicted  the  exact  final  EAC.  The  following  fonnula  further 
illustrates  this  method: 


%  Error  — 


( EACpinai  EACPredicted) 


EAC  p; 


*  100 


Final 


(4) 


Once  the  error  percentages  are  formulated  for  each  contract,  the  averages  at  each 
completion  point  are  calculated  to  detennine  the  overall  accuracy  of  the  model. 
Accuracy  percentages  are  assessed  for  predictability  of  the  program  level  final  reported 
EAC. 

Phase  Two 

Phase  two  of  the  methodology  is  slightly  more  open-ended  than  phase  one.  The 
purpose  of  this  phase  is  to  reveal  commonalities  amongst  contracts  that  are  indicative  of 
cost  growth.  In  this  phase,  each  contract  is  analyzed  to  determine  which  elements  found 
at  the  lower  levels  are  driving  EAC  cost  growth.  DC  ARC’S  CPR  Viewer  and  Microsoft 
Excel  are  the  primary  tools  utilized  to  conduct  the  assessment  of  cost  drivers.  In 
addition,  the  level  two  and  three  activities  are  explored  to  determine  which  common 
activities  and  elements  exists  between  the  different  contracts  that  may  be  explanatory  in 
terms  of  EAC  growth  and  changes.  The  commonalities  are  then  further  assessed  to 
detennine  the  extent  of  the  relationship  between  changes  in  the  EAC  and  changes  in  the 
WBS  activity. 
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In  hopes  of  finding  additional  trends,  occurrences  of  EAC  cost  growth  exceeding 
either  5%  or  10%,  dependent  upon  the  contracts  total  cost  growth,  in  subsequent 
reporting  periods  is  researched  to  detennine  which  lower  level  WBS  activities  may  have 
driven  the  increase.  Inferences  are  made  by  assessing  the  timing  and  magnitude  of  the 
excessive  cost  growth  periods.  In  addition,  contracts  are  assessed  to  identify  the  WBS 
activities  that  experienced  the  largest  EAC  growth  in  tenns  of  dollar  figure  and  percent 
growth.  Following  these  steps,  conclusions  are  drawn  based  on  trends  and  patterns  seen 
in  the  dataset. 

Summary 

In  summary,  this  chapter  provided  the  sources  of  the  sample  data  and  the 
methodological  approaches  taken  to  address  the  research  question.  Characteristics  of  the 
sample  data,  formulas  used,  and  research  steps  were  presented.  Finally,  this  chapter 
explained  the  two  phased  approach  to  provide  background  on  the  structural  framework 
for  this  research  effort.  Chapter  4  presents  the  results  of  the  discussed  methodological 
approaches. 
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IV.  Analysis  and  Results 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  provide  the  results  of  the  methodology  outlined 
in  Chapter  3.  The  results  are  presented  in  a  consistent  manner  as  the  methodology;  first, 
phase  one  results  are  presented  and  then  phase  two  results  are  given.  Phase  one  results 
detail  the  findings  based  on  WBS  level  and  using  the  variations  of  the  BCWP  model. 
Phase  two  provides  a  detailed  analysis  of  each  of  the  nine  contracts  and  then  determines 
if  revealed  commonalities  amongst  programs  and  contracts  are  indicative  of  cost  growth. 

Phase  One  Results 

The  results  of  phase  one  examine  the  differences  in  outcomes  based  on  using  data 
from  the  different  WBS  levels.  The  BCWP  model  distributes  traits  consistent  with 
previous  research  at  WBS  level  one.  The  model  is  on  average  16%  more  accurate  than 
the  EAC  Composite  and  18%  more  accurate  than  the  reported  EAC.  These  averages  are 
derived  from  assessing  the  model  at  each  completion  point  from  20%  to  70%  and  taking 
the  average  of  the  six  calculations.  At  the  60%  completion  point,  the  model  levels  out  in 
terms  of  accuracy.  In  fact,  the  average  error  is  1%  less  accurate  at  the  70%  completion 
point  than  the  60%  completion  point;  this  seems  counterintuitive  to  the  idea  that  as  a 
program  matures,  you  can  predict  the  final  costs  more  accurately.  These  results  are 
attributed  to  the  fact  that  the  BCWP  model  does  not  have  a  built  in  mechanism  to  account 
for  the  reduced  spending  seen  at  the  end  of  the  contracts  in  the  dataset.  Hence,  the  model 
over  estimates  for  all  five  contracts  at  the  70%  completion  point.  Figure  6  displays  the 
full  detail  of  the  level  one  BCWP  model  results. 
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Figure  6.  Level  1  Summary 


The  BCWP  model  at  level  two  is  on  average  17%  better  than  the  EAC  Composite 


and  19%  better  than  the  reported  EAC.  As  one  would  expect,  the  model  improves  at  each 


subsequent  completion  percentage  point.  The  results  are  fairly  consistent  with  those  seen 


at  level  one  and  are  seen  in  Figure  7. 
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Figure  7.  Level  2  Summary 
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The  BCWP  model  at  level  three  is  on  average  17%  better  than  the  EAC 
Composite  and  19%  better  than  the  reported  EAC.  The  results  are  fairly  consistent  with 


those  seen  at  levels  one  and  two,  and  are  seen  in  Figure  8. 
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Figure  8.  Level  3  Summary 

Although  there  is  not  a  large  difference  between  the  three  WBS  levels,  level  one 
is  slightly  more  accurate  than  levels  two  and  three.  A  comparison  of  levels  one  and  two 
is  seen  in  Figure  9.  There  are  no  practical  differences  between  levels  two  and  three. 
Minor  differences  between  these  two  levels  can  likely  be  attributed  to  accounting  errors 
within  the  DCARC  EVM  data.  Level  two  and  three  comparison  results  are  shown  in 
Figure  10. 

The  results  of  testing  multiple  models  reveal  interesting  findings.  No  one  model 
performs  the  best  throughout  the  various  levels  of  contract  completion  percentage.  As  a 
reminder,  Model  1  is  the  original  BCWP  model  which  utilized  a  regression  based  bum 
rate  and  earned  schedule;  Model  2  is  the  first  variation  which  utilizes  actual  schedule  in 
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place  of  earned  schedule;  Model  3  utilizes  a  monthly  average  burn  rate  and  earned 
schedule;  and  Model  4  utilizes  both  a  monthly  average  burn  rate  and  an  actual  schedule. 
Model  1  has  an  average  error  rate  of  29%.  This  model  performs  the  best  in  the  early 
stages  of  the  contract.  At  the  20%  and  30%  completion  points,  Model  1  outperforms  all 
other  models.  Model  2  has  an  average  error  rate  of  3 1%.  It  displays  similar  results  to 
Model  1  throughout  all  of  the  contract  percentage  completion  stages.  Model  2  is  slightly 
less  accurate  than  Model  1,  and  it  does  not  perform  the  best  at  any  completion  points. 
Model  3  has  an  average  error  rate  of  30%.  In  the  early  stages  of  the  contract,  it  does  not 
perfonn  well  and  has  an  average  error  rate  of  nearly  50%  at  the  20%  and  30%  completion 
points.  After  these  completion  points,  Model  3  out  performs  all  other  models.  Model  4 
has  an  average  error  rate  of  33%.  It  displays  similar  results  to  Model  3;  however,  Model 
4  is  slightly  less  accurate  than  Model  3  throughout  the  contract  percentage  completion 


stages.  It  does  not  perform  the  best  at  any  completion  points. 
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Figure  9.  Level  1  &  2  Comparison 
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Level  2  vs  Level  3 
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Figure  10.  Level  2  &  3  Comparison 

The  results  of  the  four  models  are  illustrated  in  Figure  1 1 .  The  results  suggest 
that  the  regression  based  burn  rate  works  the  best  early  on  in  the  contract’s  completion 
stage;  hence,  why  Models  1  and  2  performed  better  in  the  beginning.  The  monthly 
average  BCWP  bum  rate  works  the  best  after  the  40%  completion  point.  The  earned 
schedule  approach  seems  to  outperform  the  actual  schedule  method  at  all  stages  of 
completion.  This  is  seen  in  the  inferior  performances  of  Model  2  and  Model  4  in 
comparison  to  Model  1  and  Model  3  respectively. 

A  deeper  assessment  of  the  data  reveals  that  the  monthly  average  burn  rate 
models,  Models  3  and  4,  are  highly  influenced  by  one  contract  -  the  Advanced  Extremely 
High  Frequency  (AEHF)  contract.  The  AEHF  contract  has  a  significant  gap  between  the 
contract  start  date  and  when  EVM  reporting  began.  The  contract  started  in  November 
2001,  but  the  first  reporting  period  did  not  occur  until  October  2002.  This  delay,  results 
in  the  data  improperly  suggesting  that  17%  of  the  total  contract  is  spent  in  the  first  month. 
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This  would  mean  that  any  formulation  method  that  utilizes  a  bum  rate  based  on  monthly 
averages,  will  most  likely  overestimate  due  to  the  appearance  of  a  17%  per  month 
spending  rate.  To  control  for  this  error,  spending  was  straight-lined  over  the  number  of 
missing  months  in  the  dataset.  Following  this  correction,  Models  3  and  4  no  longer 
display  an  extremely  large  error  percentage  early  on  in  the  contract.  Model  3 
outperforms  all  models  in  nearly  all  stages  of  completion  percentage.  These  results  are 
found  in  Figure  12.  The  results  suggest  that  a  monthly  average  BCWP  burn  rate  is 
superior  to  the  regression  based  approach.  This  is  seen  in  the  comparison  of  Models  1 
and  3,  and  the  comparison  of  Models  2  and  4. 
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Figure  11.  Model  Comparisons 
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Figure  12.  Modified  Model  Comparisons 


Phase  Two  Results 

The  results  of  phase  two  present  an  in-depth  analysis  of  each  of  the  nine 
programs.  The  WBS  lower  level  composition  for  each  contract  was  vastly  dissimilar. 
Due  to  the  vast  number  of  differences  found  at  each  subsequent  WBS  level,  a  stronger 
emphasis  is  placed  on  the  analysis  of  WBS  level  two  activities  in  hopes  of  findings 
similar  elements  across  the  contracts.  Not  only  were  the  naming  conventions  unique  to 
each  contract,  but  the  number  of  elements  found  at  each  level  varied  greatly  as  well. 
Figure  13  displays  the  variation  in  the  number  of  lower  level  elements. 
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Figure  13.  WBS  Elements  by  Contract 

AEHF 

The  AEHF  contract  is  an  agreement  with  Lockheed  Martin  for  the  purchase  of  six 
communication  satellites.  It  is  composed  of  four  level  two  activities  and  seven  level 
three  activities;  however,  the  traditional  military  standard  space  WBS  level  two  naming 
convention  does  not  occur  until  WBS  level  four  and  carries  out  through  level  seven.  For 
this  reason,  levels  four  and  five  are  treated  as  levels  two  and  three  respectively  for  this 
portion  of  the  analysis.  The  AEHF  contract  exhibited  a  total  EAC  cost  growth  of  147% 
over  its  duration.  This  cost  growth  was  incited  by  several  periods  of  large  cost  growth. 

In  an  effort  to  discover  what  activities  may  have  stimulated  this  cost  growth,  any  growth 
exceeding  10%  from  one  reporting  period  to  the  next  was  further  investigated  by 
assessing  which  elements  of  the  program  experienced  representative  changes;  this 
occurred  on  four  occasions.  In  all  occasions,  there  is  no  immediate  representative  effect 
found  within  the  lower  level  cost  elements.  Instead,  the  level  one  summary  level 
element,  Undistributed  Budget  (UB),  is  the  first  to  experience  an  immediate  increase. 
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Over  the  subsequent  months,  the  newly  allocated  budget  is  distributed  amongst  the 
specific  WBS  activities. 

The  first  large  cost  growth  increase  occurs  in  October  2004  at  the  50% 
completion  point.  The  main  level  two  activities  that  increase  in  EAC  resulting  from  this 
growth  are  1.0  Space  Vehicle,  3.0  Systems  Engineering  and  Program  Management 
(SEPM),  and  7.0  Operations  and  Support  (O&S).  The  second  large  increase  occurs  in 
January  2006  at  the  60%  completion  point.  The  main  level  two  activity  that  increases 
from  this  growth  is  1.0A  Space  Vehicle.  The  third  large  increase  occurs  in  October  2008 
at  the  80%  completion  point.  The  main  level  two  activities  that  increase  from  this  growth 
are  1.0  Space  Vehicle,  1.0A  Space  Vehicle,  and  1.0C  Space  Vehicle,  and  3.0  SEPM.  The 
last  increase  exceeding  10%  occurs  in  November  2011  at  the  90%  completion  point.  The 
main  level  two  activity  that  increases  from  this  growth  is  7.0  O&S.  Figure  14  displays 
the  EAC  cost  growth  of  the  total  AEHF  contract  and  several  key  WBS  level  two 
activities  contributing  to  this  growth.  Following  the  growth  of  the  key  activities 
displayed  in  Figure  14,  WBS  level  two  activity  1.0A  Space  Vehicles  becomes  the 
primary  driver  for  EAC  cost  growth. 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  AEHF  contract  are 
the  activities  associated  with  the  actual  satellite  space  vehicles.  The  1.0  Space  Vehicles 
activity  incurs  $1.5B  and  94%  in  EAC  cost  growth.  SEPM  also  appears  to  be  indicative 
of  EAC  cost  growth.  This  WBS  activity  incurs  $243M  and  97%  in  EAC  cost  growth. 
Results  of  the  greatest  contributors  to  EAC  cost  growth  for  WBS  level  two  are  found  in 
Table  8  and  Table  9.  The  “EAC  Change”  represents  the  total  growth  in  EAC  for  the 
activity,  and  the  “%  Total  EAC”  represents  the  percentage  of  the  final  EAC  that  the 
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activity  accounts  for.  Table  9  is  not  all  inclusive  due  to  the  fact  that  some  activities  are 
not  present  in  the  early  stages  of  the  contract.  These  activities  would  have  a  base  EAC 
amount  of  zero  and  result  in  an  undefined  cost  growth.  Attempts  aimed  at  using  the  first 
reported  EAC  results  in  impractical  results.  For  example,  the  first  reported  EAC  for  3.0 
Intersegment  SEPM  is  $400,  while  the  final  reported  EAC  is  nearly  $57M.  This  results 
in  an  EAC  growth  of  over  14M  percent.  It  is  probably  unrealistic  to  assume  that 
contractors  believed  this  activity  would  require  only  $400.  Instead,  the  early  estimates 
for  many  of  the  added  activities  appear  to  be  placeholders  used  until  additional  funding 
became  available.  This  was  a  common  theme  found  amongst  many  of  the  nine  contracts, 


Similar  analysis  conducted  for  level  three  activities  can  be  found  in  Appendix  B. 


Figure  14.  AEHF  EAC  Cost  Growth 
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Table  8.  AEHF  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.0:  SPACE  \THICLE 

SI  ,46 1,250 

48°  o 

1.0 A:  SPACE  \THICLE 

S658.606 

11% 

7.0:  OPERATIONS  &  SUPPORT 

S426.446 

7% 

2.0:  MISSION  CONTROL  SYSTEM 

S357.167 

14% 

3.0:  INTERSEGMENT  SYS  ENG  PGM  MGMT 

S243.287 

o 

o 

GO 

Table  9.  AEHF  Level  2  %  EAC  Change 


WBS  Activity 

EAC  Change 

%  Total  EAC 

3.0:  INTERSEGMENT  SYS  ENG  PGM  MGMT 

97% 

8% 

6.0:  INTERSEGMENT  SYSTEM  LE\TL  I&T 

95% 

4% 

1.0:  SPACE  VEHICLE 

94% 

48% 

2.0:  MISSION  CONTROL  SYSTEM 

66% 

14% 

FAB-T 

The  FAB-T  contract  is  an  agreement  with  Boeing  for  satellite  communications 
terminals  designed  for  airborne,  ground-fixed,  and  ground-transportable  applications.  It 
is  composed  of  eight  level  two  activities  and  twenty  level  three  activities.  The  FAB-T 
contract  exhibited  an  EAC  cost  growth  of  77%  over  its  duration.  There  were  two  back- 
to-back  reporting  periods  around  the  70%  completion  point  in  which  the  FAB-T  contract 
experience  EAC  growth  exceeding  10%.  The  first  occurred  in  February  2010,  but  no 
level  two  activities  saw  any  representative  increases.  The  only  notable  increase  was  to 
the  level  one  UB  account.  The  second  large  increase  occurred  in  March  2010.  This 
increase  resulted  in  a  representative  growth  in  nearly  all  level  two  activities.  Figure  15 
displays  the  EAC  cost  growth  of  the  total  FAB-T  contract  and  several  key  WBS  level  two 
activities  contributing  to  this  growth. 
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EAC 

1.03:  SEPM 


•  —  1.04:  System  T&E 

—  —  1.01:  FAB-T  PMP 


Figure  15.  FAB-T  EAC  Cost  Growth 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  FAB-T  contract  are 
the  Prime  Mission  Product  (PMP)  and  SEPM.  PMP  incurred  an  EAC  cost  growth  of 
$500M  and  111%;  SEPM  incurred  a  cost  growth  of  $  126M  and  39%.  Although  the 
dollar  amount  was  not  as  significant,  the  Test  and  Evaluation  activity  experienced  a 
138%  EAC  cost  growth.  Results  of  the  greatest  contributors  to  EAC  cost  growth  for 
WBS  level  two  are  found  in  Table  10  and  Table  11.  Similar  analysis  conducted  for  level 
three  activities  can  be  found  in  Appendix  B. 
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Table  10.  FAB-T  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.01:  FAB-T  PRIME  MISSION  PRODUCT  (PMP) 

$500,381 

64% 

1.03:  SYSTEMS  ENGINEERING  PROGRAM  MANAGEMENT 

S125,870 

30% 

1.04:  SYSTEM  (GOVERNMENTAL)  TEST  AND  EVALUATION 

S36.284 

4% 

1.05:  TRAINING 

$3,212 

1% 

1.09:  OPERATIONAL  SITE  ACTIVATION  (GROUND) 

$1,165 

0.4% 

Table  11.  FAB-T  Level  2  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.04:  SYSTEM  (GOVERNMENTAL)  TEST  AND  EVALUATION 

138% 

4% 

1.01:  FAB-T  PRIME  MISSION  PRODUCT  (PMP) 

111% 

64% 

1.03:  SYSTEMS  ENGINEERING  /  PROGRAM  MANAGEMENT 

39% 

30.0% 

1.05:  TRAINING 

34% 

1% 

1.09:  OPERATIONAL/SITE  ACTIVATION  (GROUND) 

22% 

0.4% 

GPS  III 

The  GPS  III  contract  is  an  agreement  with  Lockheed  Martin  for  satellite 
communications.  It  is  composed  of  five  level  two  activities  and  eighteen  level  three 
activities.  Three  of  the  level  two  activities  are  not  populated  with  data  throughout  the 
contract.  The  GPS  III  contract  exhibited  an  EAC  cost  growth  of  52%  over  its  duration. 
The  contract  did  not  experience  any  cost  growth  exceeding  10%  from  one  reporting 
period  to  the  next;  however  there  were  three  instances  that  cost  growth  exceeded  5%.  In 
all  instances,  the  level  two  1 .2  GPS  III  activity,  which  accounts  for  over  90%  of  the 
contract,  experienced  representative  growth  increases.  The  first  instance  occurred  in 
December  2010  around  the  50%  completion  point.  The  level  one  UB  account  and  the 
level  two  1.2  GPS  III  activity  had  the  largest  increases.  The  second  increase  occurred  in 
May  2012  around  the  70%  completion  point.  This  increase  also  resulted  in  representative 
increases  to  the  UB  account  and  the  1 .2  GPS  III  activity.  The  third  increase  occurred  in 
December  2013  around  the  84%  completion  point.  This  increase  resulted  in  a  direct 
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increase  to  the  1.2  GPS  III  activity.  Figure  16  displays  the  EAC  cost  growth  of  the  total 
GPS  III  contract  and  both  WBS  level  two  activities  contributing  to  this  growth. 


•EAC 

1.1:  SEIT/PM 
1.2:  GPS  III 


Figure  16.  GPS  III  EAC  Cost  Growth 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  GPS  III  contract  are 
the  Satellite  and  SEPM  activities.  The  GPS  Satellite  activity  incurred  an  EAC  cost 
growth  of  $649M  and  55%;  SEPM  grew  by  $35M  and  26%.  Results  of  the  GPS  III  EAC 
cost  growth  for  WBS  level  two  activities  are  found  in  Table  12  and  Table  13.  Similar 
analysis  conducted  for  level  three  activities  can  be  found  in  Appendix  B. 


Table  12.  GPS  III  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.2:  GPS  m  -  GLOBAL  POSITIONING  SATELLITE  ID 

S649.144 

91% 

1.1:  SEIT  /  PM  AND  OTHER  COMMON  ELEMENTS 

S35.201 

9% 
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Table  13.  GPS  III  Level  2  %  EAC  Change 


WBS  ACTIVITY 


EAC  Change 

55% 

26% 


0  o  Total  EAC 

91% 

9% 


1.2:  GPS  III  -  GLOBAL  POSITIONING  SATELLITE  m 
1.1:  SEIT  /  PM  AND  OTHER  COMMON  ELEMENTS 


MUOS 


The  MUOS  contract  is  an  agreement  with  Lockheed  Martin  for  five  satellites 
aimed  at  significantly  improving  ground  communications.  It  is  composed  of  eight  level 
two  activities  and  twenty-one  level  three  activities.  Four  of  the  level  two  activities  are 
not  populated  with  data  throughout  the  contract.  The  MUOS  contract  exhibited  an  EAC 
cost  growth  of  96%  over  its  duration.  There  were  two  periods  of  large  cost  growth 
exceeding  10%  with  the  MUOS  contract.  Both  occurred  after  the  contract  reached  the 
70%  completion  point.  The  first  occurred  in  November  2008.  There  is  an  immediate 
increase  seen  in  the  level  one  UB  account  and  the  level  two  1000  Satellite  activity.  In  the 
following  reporting  period,  sharp  increases  are  seen  in  both  the  3000  Integrated  Ground 
and  4000  SEPM  level  two  activities.  The  second  large  increase  occurs  in  January  2009. 
This  increase  is  primarily  driven  by  an  increase  to  the  3000  Integrated  Ground  activity. 
Figure  17  displays  the  EAC  cost  growth  of  the  total  MUOS  contract  and  several  key 
WBS  level  two  activities  contributing  to  this  growth. 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  MUOS  contract  are 
the  Satellite,  Integrated  Ground,  and  SEPM.  The  SA  activity  incurs  $787M  and  122%  in 
EAC  cost  growth;  the  IG  activity  incurs  $55 1M  and  74%;  and  the  SEPM  activity  incurs 
$208M  and  75%.  The  System  Support  activity  experiences  over  250%  in  EAC  cost 
growth,  but  the  costs  are  not  as  significant  as  other  activities.  Results  of  the  greatest 
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contributors  to  EAC  cost  growth  for  WBS  level  two  are  found  in  Table  14  and  Table  15. 
Similar  analysis  conducted  for  level  three  activities  can  be  found  in  Appendix  B. 


EAC 

4000:  SEPM 

3000:  Integrated  Ground 

1000:  Satellite 


Figure  17.  MUOS  EAC  Cost  Growth 


Table  14.  MUOS  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1000:  SATELLITE  (SA) 

$787,195 

43% 

3000:  INTEGRATED  GROUND  (IG) 

$551,115 

39% 

4000:  SYSTEM  ENGINEERING  PROGRAM  MANAGEMENT 

S208A76 

15% 

5000:  SYSTEM  SUPPORT  AND  SUPPORTABILITY  (ILS) 

S86,046 

4% 

Table  15.  MUOS  Level  2  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

5000:  SYSTEM  SUPPORT  AND  SUPPORTABILITY  (ILS) 

256% 

4% 

1000:  SATELLITE  (SA) 

122% 

43% 

4000:  SYSTEM  ENGINEERING  PROGRAM  MANAGEMENT 

75% 

15% 

3000:  INTEGRATED  GROUND  (IG) 

74% 

39% 
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WGS 


The  WGS  contract  is  an  agreement  with  Boeing  for  a  constellation  of  military 
communications  satellites.  It  is  composed  of  one  activity  at  the  second  WBS  level  and 
nine  activities  at  the  third  WBS  level;  however,  the  traditional  military  standard  space 
WBS  level  two  naming  convention  does  not  occur  until  WBS  level  three  and  carries  out 
through  level  nine.  For  this  reason,  levels  three  and  four  are  treated  as  levels  two  and 
three  respectively  in  this  portion  of  the  analysis.  Three  of  the  respective  level  two 
activities  are  not  populated  with  data  throughout  the  contract.  In  addition,  three  other 
activities  do  not  experience  positive  EAC  cost  growth.  The  WGS  contract  exhibited  an 
EAC  cost  growth  of  207%  over  its  duration.  This  cost  growth  was  incited  by  three 
occurrences  of  cost  growth  exceeding  10%. 

In  all  three  instances,  the  UB  account  instantly  exhibits  a  representative  increase, 
followed  by  the  distribution  of  funding  to  the  1.1  Satellite  Vehicles  and  1.2  SEPM  WBS 
level  two  activities.  The  first  occurs  in  January  2007  around  the  16%  completion  point. 
The  funding  behind  this  increase  remains  in  the  UB  account  for  six  reporting  periods 
before  it  is  distributed  to  the  1.1  Satellite  Vehicles  and  1.2  SEPM  WBS  level  two 
activities.  The  second  occurs  in  January  2008  around  the  34%  completion  point.  This 
funding  remains  in  the  UB  account  for  two  reporting  periods  before  it  is  distributed  to  the 
same  level  two  accounts.  The  third  large  cost  increase  occurs  in  December  2008  around 
the  49%  completion  point.  This  funding  results  in  increases  to  the  same  level  two 
accounts  spread  over  the  next  three  reporting  periods.  Figure  18  displays  the  EAC  cost 
growth  of  the  total  WGS  contract  and  several  key  WBS  level  two  activities  contributing 
to  this  growth. 
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Figure  18.  WGS  EAC  Cost  Growth 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  WGS  contract  are  the 
Satellite  Vehicle  (SAV)  and  SEPM  activities.  The  SAV  activity  incurs  $569M  and  227% 
in  EAC  cost  growth;  the  SEPM  activity  incurs  $101M  and  143%  in  cost  growth.  The 
Command  and  Control  activity  incurs  133%  EAC  cost  growth,  but  the  dollar  amount  is 
relatively  small.  Results  of  the  greatest  contributors  to  EAC  cost  growth  for  WBS  level 
two  are  found  in  Table  16  and  Table  17.  Similar  analysis  conducted  for  level  three 
activities  can  be  found  in  Appendix  B. 


Table  16.  WGS  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.1:  SATELLITE  VEHICLE 

S569.024 

82% 

1.2:  SYSTEM  LEVEL  SYS  ENGRG  PROG  MGMT 

S101.044 

17% 

1.5:  COMMAND  &  CONTROL  SUBSYSTEM 

S3.383 

1% 

1.7:  ENGINEERING  SUPPORT/STUDIES 

S818 

0.1% 

1.9:  DATA 

S51 

0.01% 
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Table  17.  WGS  Level  2  %  EAC  Change 


WBS  ACTIVITY 


EAC  Change 

227% 

143% 

133% 


0  o  Total  EAC 

82% 

17% 

1% 


1.1:  SATELLITE  VEHICLE 

1.2:  SYSTEM  LEVEL  SYS  ENGRG  PROG  MGMT 

1.5:  COMMAND  &  CONTROL  SUBSYSTEM 


EELV 


The  EELV  contract  is  an  agreement  with  the  United  Launch  Alliance  to  improve 
U.S.  space  launch  capabilities.  It  is  composed  of  six  level  two  activities  and  seventy- 
seven  level  three  activities.  At  the  time  of  the  data  collection  point  cut-off,  the  EELV 
contract  exhibited  an  EAC  cost  decline  of  3%  and  was  88%  complete.  In  addition,  no 
level  two  activities  changed  by  more  than  10%,  and  the  level  three  activities  that  did 
experience  significant  percentage  change  were  insignificant  in  dollar  amount.  As  of 
January  2015,  the  EELV  contract  is  at  55%  complete,  and  it  has  an  estimated  EAC 
growth  of  over  100%.  All  populated  level  two  WBS  activities  experience  greater  than 
73%  in  EAC  cost  growth.  The  SEPM  activity  is  currently  estimated  to  incur  over  $300M 
in  EAC  growth  which  is  greater  than  twice  the  amount  of  any  other  activity.  Although 
the  EELV  contract  is  mentioned  throughout  this  research,  it  is  not  utilized  to  draw 
conclusions  and  make  recommendations  due  to  its  revised  completion  percentage. 

NAVSTAR 

The  NAVSTAR  contract  is  an  agreement  with  Raytheon  Company  for  next- 
generation  global  positioning  receivers.  It  is  composed  of  one  level  two  activity  and 
seven  level  three  activities.  Although  the  naming  convention  is  atypical,  the  standard 
level  two  activities  are  represented  as  level  three  in  this  contract;  for  this  reason,  levels 
three  and  four  are  treated  as  levels  two  and  three  respectively.  The  NAVSTAR  contract 
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exhibited  an  EAC  cost  growth  of  107%  over  its  duration.  This  cost  growth  was  driven  by 
two  distinct  periods  of  growth  exceeding  10%.  The  first  instance  occurred  in  December 
2011  at  the  67%  completion  point.  Both  the  15:  SEPM  and  13:  Common  Components 
level  activities  experienced  immediate  EAC  growth.  The  second  instance  occurred  the 
following  month,  January  2012,  at  the  69%  completion  point.  The  funding  for  this 
increase  is  distributed  to  the  level  one  UB  account  prior  to  getting  distributed  to  the  12: 
MUE  PMP#2,  13:  Common  Components,  and  15:  SEPM  level  two  activities  the 
following  month.  Figure  19  displays  the  EAC  cost  growth  of  the  total  NAVSTAR 


contract  and  several  key  WBS  level  two  activities  contributing  to  this  growth. 


EAC  Growth 

^  ^  13:  Common  Components 

—  —  12:  MUE  PMP  #2  GB-GR 


Figure  19.  NAVSTAR  EAC  Cost  Growth 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  NAVSTAR  contract 
are  the  Common  Components,  PMP#2,  and  SEPM  activities.  The  Common  Components 
activity  incurs  $16.6M  and  115%  in  EAC  cost  growth;  PMP#2  incurs  $15.5M  and  98%; 
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and  SEPM  incurs  $14M  and  138%.  Results  of  the  greatest  contributors  to  EAC  cost 
growth  for  WBS  level  two  are  found  in  Table  18  and  Table  19.  Similar  analysis 
conducted  for  level  three  activities  can  be  found  in  Appendix  B. 


Table  18.  NAVSTAR  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

13:  COMMON  COMPONENTS 

S16.635 

27% 

12:  MUE  PRIME  MISSION  PRODUCT  (PMP)  =2  GB-GR 

S15.548 

27% 

15:  SE'PM 

SI  3,904 

21% 

11:  MUE  PRIME  MISSION  PRODUCT  (PMP)  #1  GRAM 

S7,966 

16% 

IS  DATA 

S62 

0.4% 

Table  19.  NAVSTAR  Level  2  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

15:  SE  PM 

138% 

21% 

13:  COMMON  COMPONENTS 

115% 

27% 

12:  MUE  PRIME  MISSION  PRODUCT  (PMP)  #2  GB-GR 

98% 

27% 

1 1:  MUE  PRIME  MISSION  PRODUCT  (PMP)  #1  GRAM 

76% 

16% 

18:  DATA 

14% 

0.4% 

GPS  OCX 

The  GPS  OCX  contract  is  an  agreement  with  Northrop  Grumman  Corporation  for 
ground  control  communication  systems.  It  is  composed  of  three  level  two  activities  and 
twenty-three  level  three  activities.  The  traditional  military  standard  space  WBS  level  two 
naming  convention  does  not  occur  until  WBS  level  three.  For  this  reason,  levels  three 
and  four  are  treated  as  levels  two  and  three  respectively  for  this  portion  of  the  analysis. 
The  GPS  OCX  contract  exhibited  an  EAC  cost  growth  of  33%  over  its  duration.  The 
contract  did  not  experience  any  cost  growth  exceeding  10%  from  one  reporting  period  to 
the  next;  however  there  were  two  instances  of  cost  growth  exceeding  5%.  In  both 
occurrences,  the  level  one  UB  account  experiences  a  representative  increase;  funds  are 
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then  distributed  to  the  1.01  OCX  SEPM  WBS  level  two  activity.  The  first  instance 
occurred  in  February  2009  around  the  76%  completion  point,  and  the  second  instance 
occurred  in  September  2009  around  the  94%  completion  point.  Figure  20  displays  the 
EAC  cost  growth  of  the  total  GPS  OCX  contract  and  several  key  WBS  level  two 
activities  contributing  to  this  growth. 

The  main  contributor  to  EAC  cost  growth  for  the  GPS  OCX  contract  is  the  SEPM 
activity.  The  SEPM  activity  incurs  $36M  and  59%  in  EAC  cost  growth.  Many  of  the 
activities  exhibiting  extremely  high  levels  of  cost  growth  are  not  practically  significant  in 
relation  to  total  contract  costs.  Results  of  the  greatest  contributors  to  EAC  cost  growth 
for  WBS  level  two  are  found  in  Table  20  and  Table  21.  Similar  analysis  conducted  for 
level  three  activities  can  be  found  in  Appendix  B. 


Figure  20.  GPS  OCX  EAC  Cost  Growth 
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Table  20.  GPS  OCX  Level  2  $  EAC  Change  ($K) 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.01:  OCX  SYSTEMS  ENGINEERING  /  PROGRAM  MGMT 

536,048 

55% 

1.18:  OCX  MISSION  INFRASTRUCTURE  S/S 

S2.239 

19% 

2.02:  GPS  SYS/SYS  RQS  VERIF,  VALADATTON 

$1,800 

1% 

1.13:  COMMAND  &  CONTROL  SUBSYSTEM  (TT&C) 

SI,  447 

7% 

1.12  GROUND  TERMINAL  SUBSYSTEMS 

Sl,371 

5% 

Table  21.  GPS  OCX  Level  2  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

2.01:  SYSTEM  INTEGRATION  DEMOS 

890% 

0.4% 

2.03:  SYSTEM  OF  SYSTEMS  TEST  SUPPORT 

466% 

0.1% 

2.02:  GPS  SYS/SYS  RQS  VERIF,  VALADATTON 

301% 

1% 

1.17:  MISSION  DATA  ANALYS  &  DISSEMINATION  S/S 

79% 

1% 

1.01:  OCX  SYSTEMS  ENGINEERING  /  PROGRAM  MGMT 

59% 

55% 

WGS  10-C 

The  WGS  10-C  contract  is  an  agreement  with  Boeing  for  the  seventh  satellite  of  a 
constellation  of  satellites.  It  is  composed  of  two  level  two  activities  and  five  level  three 
activities.  The  WGS  10-C  contract  exhibited  an  EAC  cost  growth  of  6%  over  its 
duration.  Although  this  level  of  cost  growth  would  be  acceptable  by  most  standards, 
analysis  is  still  conducted  on  any  sharp  increases.  There  was  one  instance  of  cost  growth 
exceeding  5%.  This  growth  occurred  in  January  2012  around  the  75%  completion  point. 
The  growth  resulted  in  an  increase  to  the  1.2  Space  Vehicle  WBS  level  two  activity. 
Figure  21  displays  the  EAC  cost  growth  of  the  total  WGC  10-C  contract  and  both  WBS 
level  two  activities  contributing  to  this  growth. 

The  main  level  two  contributors  to  EAC  cost  growth  for  the  WGS  10-C  contract 
are  the  SEPM  and  Space  Vehicle  activities.  These  are  also  the  only  two  activities  at  level 
two.  The  SEPM  activity  incurs  a  $5M  and  1 1%  EAC  cost  growth;  Space  Vehicle  incurs 
$3.7M  and  nearly  4%.  Results  of  these  contributors  to  EAC  cost  growth  for  WBS  level 
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two  are  found  in  Table  22  and  Table  23.  Similar  analysis  conducted  for  level  three 
activities  can  be  found  in  Appendix  B. 


EAC 

1.1:  System  Level  SEIT/PM 
1.2:  Space  Vehicle 


Figure  21.  WGS  10-C  EAC  Cost  Growth 


Table  22.  WGS  C-10  Level  2  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1:  SYSTEM  LEVEL  SEIT/PM 

S5.075 

32% 

1.2:  SPACE  VEHICLE 

S3.727 

68% 

Table  23.  WGS  C-10  Level  2  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1:  SYSTEM  LEVEL  SEITPM 

11% 

32% 

1.2:  SPACE  VEHICLE 

4% 

68% 
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Discussion 


The  only  true  commonality  amongst  cost  drivers  for  the  nine  space  contracts  is 
the  SEPM  activity.  Based  on  the  findings,  it  would  seem  that  SEPM  is  seemingly  a 
significant  factor  in  determining  EAC  cost  growth.  The  fault  in  this  logic  is  that  SEPM  is 
commonly  estimated  as  a  factor  of  either  total  costs  or  the  PMP  (AFCAA,  2007). 
Nonetheless,  analysis  is  conducted  to  determine  if  there  are  certain  traits  involving  SEPM 
that  may  dictate  EAC  cost  growth. 

Table  24  displays  the  descriptive  statistics  of  the  eight  contracts  and  the  SEPM 
WBS  activity.  Data  is  gathered  at  each  monthly  reporting  period  for  every  contract,  and 
the  percentage  of  SEPM  is  calculated.  The  findings  suggest  that  there  is  not  a  positive 
correlation  between  the  average  amount  of  SEPM  and  the  EAC  cost  growth.  It  would 
almost  seem  that  the  fewer  amounts  spent  on  SEPM,  the  higher  the  EAC  cost  growth. 

The  standard  deviations  of  the  monthly  SEPM  percentages  are  relatively  low,  and  there  is 
a  strong  linear  dependency  between  the  percent  change  of  SEPM  and  percent  change  of 
total  EAC.  These  findings  would  support  the  thought  of  SEPM  calculated  as  a 
percentage  of  total  costs. 


Table  24.  SEPM  Descriptive  Statistics 


Program  Name 

%  Complete 

EAC  Cost  Growth 

Avg  Monthly 
SEPM  % 

Std  Dev 

SEPM  vs  Tot  EAC 
(Pearson's  R) 

AEHF 

99% 

147% 

8% 

0.8% 

0.98 

FAB-T 

92% 

77% 

33% 

2.8% 

0.99 

GPSm 

87% 

52% 

9% 

0.7% 

0.95 

MUOS 

97% 

96% 

14% 

0.9% 

0.98 

WGS 

100% 

207% 

18% 

2.1% 

0.97 

NAVSTAR  GPS 

100% 

107% 

18% 

1.4% 

0.99 

GPS  OCX 

100% 

33% 

49% 

3.4% 

0.95 

WGS  10-C 

100% 

6% 

32% 

1.0% 

0.82 
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In  addition  to  assessing  the  SEPM  activity,  attempts  are  made  to  group  certain 
level  two  WBS  activities  together  by  calling  them  the  Prime  Mission  Product  (PMP)  of 
the  contract.  By  grouping  activities  in  this  manner,  the  goal  is  to  discover  any  trends  that 
may  exist  between  the  activities’  cost  growth  and  the  programs’  overall  cost  growth.  In 
most  cases,  it  is  either  explicitly  stated  or  relatively  intuitive  in  determining  what  might 
be  called  the  PMP.  For  the  purpose  of  this  portion  of  the  analysis,  the  PMP  of  the 
contract  is  defined  as  the  WBS  level  activity  that  accounts  for  the  largest  percentage  of 
the  total  contract  costs.  The  one  exception  to  this  definition  is  the  GPS  OCX  contract,  in 
which  the  SEPM  activity  represents  the  highest  percentage  of  costs.  This  exception  is 
made  due  to  the  evidence  suggesting  that  SEPM  is  often  computed  as  a  factor  of  another 
cost. 

The  PMP  for  each  contract  is  assessed  from  the  20%  to  70%  completion  points  to 
detennine  if  trends  exists  that  are  representative  of  EAC  cost  growth.  The  findings 
suggest  that  there  are  no  common  traits  between  the  mid-acquisition  EAC  cost  growth  of 
the  PMP  and  the  overall  cost  growth  of  the  contract.  This  assessment  is  conducted  for 
the  total  contract  mid-acquisition  EAC  cost  growth  as  well.  In  both  assessments,  there 
does  not  appear  to  be  any  consistent  themes.  Some  contracts  experience  PMP  and  total 
EAC  cost  growth  early  on,  while  others  do  not.  In  either  case,  the  data  do  not  present  a 
strong  argument  to  suggest  that  mid-acquisition  growth  will  be  indicative  of  overall  EAC 
growth.  Results  of  these  assessments  are  found  in  Appendix  C. 
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Summary 

In  summary,  this  chapter  provided  the  results  of  the  aforementioned  methodology 
taken  to  answer  the  research  and  investigative  questions.  Analyses  were  presented  from 
both  phase  one  and  phase  two  of  the  methodology.  Many  of  the  implications  of  these 
results  were  presented  and  briefly  discussed  in  this  chapter;  however  Chapter  5  presents 
the  overarching  conclusions  and  closing  thoughts  for  the  research. 
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V.  Conclusions  and  Recommendations 


Introduction 

This  thesis  provides  examination  of  the  premise  that  lower  level  WBS  data  can 
improve  assessments  made  on  contract  perfonnance  midway  through  its  completion. 

Past  research  suggests  that  further  investigation  is  needed  to  assess  the  distinct  features  of 
specific  commodity  types  at  various  completion  percentages.  This  thesis  explores  some 
of  those  distinctions  within  the  space  commodity.  Analysis  is  conducted  using  WBS 
level  two  and  three  data  in  hopes  of  discovering  methods  that  produce  more  accurate 
Estimate  at  Complete  (EAC)  figures  and  finding  out  which  activities  act  as  potential 
triggers  for  EAC  cost  growth. 

Conclusions  of  Research 

In  attempting  to  answer  the  question  of  whether  or  not  lower  level  WBS  data  can 
improve  the  overall  accuracy  of  EVM  based  prediction  methods,  several  findings  were 
discovered.  From  the  first  portion  of  the  analysis,  we  learned  that  lower  level  EVM  data 
did  not  produce  more  accurate  EAC  figures.  Our  analysis  focused  on  summing  up  the 
activities  of  the  lower  levels  and  using  those  totals  in  a  BCWP  model.  As  a  result,  one 
conclusion  that  can  be  drawn  with  little  debate  is  that,  when  predicting  program  EACs, 
the  use  of  lower  level  WBS  data  totals  in  models  that  are  associated  with  some  sort  of 
bum  rate  calculation  will  produce  estimates  that  are  not  as  accurate  as  the  program  level 
data.  This  conclusion  is  driven  by  the  fact  that  certain  summary  level  accounts  do  not  get 
distributed  down  to  lower  WBS  levels. 
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Conclusions  about  the  perfonnance  of  the  BCWP  model  itself  can  be  drawn  as 
well.  As  mentioned  in  Chapter  4,  the  model’s  accuracy  seems  to  level  out  at  the  60% 
completion  point.  This  leveling  out  is  the  result  of  the  model’s  tendency  to  overestimate 
final  costs  particularly  in  the  latter  stages  of  the  contract.  This  discovery  begs  the 
question  about  the  validity  of  the  model  altogether.  In  other  words,  if  all  the  model  does 
is  produce  an  estimate  that  is  higher  than  current  methods,  then  it  would  make  sense  that 
it  is  generally  more  accurate.  It  is  well  known  that  DoD  contracts  are  historically  behind 
schedule  and  over  budget.  In  the  infrequent  instances  where  a  contract  is  not  over 
budget,  the  BCWP  model  may  end  up  being  no  more  accurate  than  the  EAC  Composite 
or  some  other  widely  accepted  calculation. 

The  second  portion  of  the  research  brought  on  many  interesting  findings.  First, 
the  level  of  variation  even  within  the  same  commodity  was  quickly  brought  to  light.  The 
use  of  the  traditional  Military  Standard  88 1C  WBS  composition  was  very  inconsistent. 

In  many  cases,  lower  WBS  levels  had  to  be  treated  as  levels  two  and  three  just  to  make 
comparisons  across  different  contracts.  The  assumption  that  the  adjusted  levels  are 
accurately  portrayed  presents  additional  limitations  to  the  findings  of  this  study.  Even 
with  the  adjustments,  level  two  and  three  activities  still  vary  greatly  between  contracts 
and  within  programs.  This  variation  led  to  no  common  themes  or  elements  being  found 
that  are  indicative  of  EAC  cost  growth.  Each  contract  seemed  to  have  its  own  unique 
story. 

An  additional  finding  was  the  frequency  utilization  of  the  program  level 
Undistributed  Budget  (UB)  account.  Many  contracts  appeared  to  use  this  account  as  a 
buffer  for  newly  acquired  funds.  The  issue  this  presents  when  conducting  analyses  is  that 
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it  becomes  difficult  to  track  lower  level  cost  drivers  due  to  the  lag  in  the  distribution  of 
the  budget.  While  funding  remains  in  the  UB  account,  there  are  positive  and  negative 
adjustments  made  at  lower  levels  that  may  or  may  not  be  related  to  the  additional  funding 
received.  Analysts  would  have  to  guess  where  the  funding  was  distributed  based  on  the 
timing  and  magnitude  of  change  found  in  the  lower  level  activities.  The  Fonnat  5  CPR 
provides  some  explanation  for  changes;  however,  the  explanations  are  not  always  easily 
interpreted,  provide  limited  details,  and  do  not  often  point  to  a  specific  WBS  activity. 

Significance  of  Research 

The  findings  presented  in  this  research  may  at  first  glance  appear  to  challenge  the 
necessity  of  having  lower  level  contract  data  altogether.  The  reality  is  that  it  may  still  be 
too  early  to  draw  a  consideration  that  drastic.  This  research  does,  however,  point  out  the 
fact  that  standardization  of  lower  level  WBS  composition  is  desperately  needed.  The 
current  methods  of  reporting  lower  level  data  do  not  seem  to  provide  any  additional 
benefits  that  cannot  be  gained  by  assessing  contracts  at  the  program  level.  Until  a  more 
uniform  approach  is  adopted,  it  may  be  difficult  for  program  managers  and  cost  analysts 
to  highlight  activities  across  multiple  contracts  that  are  effecting  total  program  costs. 

The  general  statement  that  states:  lower  WBS  levels  should  sum  up  to  the  level 
immediately  preceding  it,  is  a  common  misconception.  This  train  of  thought  stems  from 
what  is  known  as  the  “100%  Rule”.  Military  Standard  88 1C  defines  this  rule  as  the 
principle  that  states  “the  next  level  of  decomposition  of  a  WBS  element  (child  level)  must 
represent  100%  of  the  work  applicable  to  the  next  higher  level  (parent  level)”  (DoD, 
2011).  Although  an  experienced  analyst  may  know  the  inherent  assumption  in  this  rule, 
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many  others  do  not.  The  assumption  made  is  that  every  parent  in  the  higher  level 
actually  has  a  child  element.  As  seen  in  this  research  with  the  UB  account,  this  is  not 
always  the  case.  If  a  higher  level  WBS  activity  does  not  have  any  lower  level  elements, 
the  sum  of  the  subsequent  level  will  be  lower  than  the  higher  level  if  this  difference  is  not 
accounted  for. 

Recommendations  for  Future  Research 

This  research  defined  model  performance  based  solely  on  accuracy  to  the  last 
reported  EAC,  but  is  accuracy  really  the  only  important  attribute?  A  well-defined  cost 
estimate  should  be  somewhat  accurate,  but  it  should  also  be  credible,  reliable,  and 
comprehensive;  the  same  should  hold  true  for  a  cost  estimating  model.  The  idea  that  the 
BCWP  model  may  simply  be  a  calculation  method  that  produces  higher  cost  estimates 
was  presented  earlier  in  this  chapter.  Additional  research  should  be  conducted  to 
determine  the  robustness  of  the  BCWP  model.  How  well  does  the  model  do  in  realms 
other  than  space?  How  well  does  the  model  perfonn  with  contracts  that  are  not  behind 
schedule  and  over  budget?  Research  should  analyze  alternative  variables  in  addition  to 
accuracy  to  measure  model  performance. 

The  assessment  of  lower  level  data  in  a  commodity  other  than  space  also  presents 
an  opportunity  for  additional  research.  Space  has  a  long  history  of  vastly  exceeding  cost 
estimates.  Maybe  a  specific  commodity  that  has  been  more  stable  in  terms  of  cost 
estimating  will  have  less  variation  at  the  lower  levels. 
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Summary 

There  is  little  doubt  that  the  Department  of  Defense’s  data  requirement  for  the 
first  three  WBS  levels  results  in  added  contractor  costs.  The  findings  of  this  research 
question  the  benefits  of  this  data  in  the  current  construct.  From  a  macro-perspective,  this 
research  also  highlights  the  difficulty  in  attempting  to  predict  EAC  cost  growth 
altogether.  As  Keller  et  al.  also  alluded  to,  perhaps  effort  should  be  redirected  from 
improving  cost  estimation  models,  and  instead  aimed  towards  improving  the  way  DoD 
management  controls  program  costs  (Keller  et  al.,  2014).  An  array  of  reasons  exists  for 
program  cost  growth,  each  having  a  unique  effect  on  the  program’s  activities,  yet,  there 
lies  an  inherent  expectation  for  cost  estimates  to  holistically  capture  those  changes  and 
still  remain  somewhat  accurate.  In  the  end,  this  expectation  may  not  lead  to  improved 
cost  estimate  accuracy  at  all;  the  result  could  very  well  be  arbitrarily  inflated  estimates 
that  are  seen  as  improvements  to  the  status  quo  based  solely  on  their  ability  to  capture  a 
portion  of  the  unpredictable  cost  growth  likely  to  be  seen. 
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Appendix  A:  Format  1  Example 


CLASSIFICATION 


COST  PERFORMANCE  REPORT 


1.  CONTRACTOR 

2.  CONTRACT 

3.  PROGRAM 

4.  REPORT  PERIOD 

a.  NAME 

US  LHC  Accelerator  Project  Office 

a.  NAME 

US  LHC 

a.  NAME 

US  LHC  Accelerator  Project 

a.  FROM  (YYMMDD) 

990201 

b.  LOCATION  (Address  and  ZIP  Code) 

P.O.  Box  500 

MS  343 

Batavia.  IL  60510 

b.  NUMBER 

1 

b.  TO  (YYMMDD) 

990228 

c.  TYPE  d.  SHARE  RATIO 

FPI  100/0  100/0 

b.  PHASE  (X one) 

”71  RDT&E  1  x  I  PRODUCTION 

5.  CONTRACT  DATA 


a.  QUANTITY 

b.  NEGOTIATED 

c.  EST.  COSTAUTH 

d.  TARGET  PROFIT/ 

e.  TARGET  PRICE 

f.  ESTIMATED 

g.  CONTRACT 

h.  ESTIMATED  CONTRACT 

COST 

UNPRICED  WORK 

FEE 

PRICE 

CEILING 

CEILING 

0/0/0 

89,417.8 

0.0 

0.0/00% 

89,417.8 

89,417.8 

110,000.0 

110,000.0 

6.  ESTIMATED  COST  AT  COMPLETION 


7.  AUTHORIZED  CONTRACTOR  REPRESENTATIVE 


a.  BEST  CASE 


b.  WORST  CASE 


c.  MOST  LIKELY 


MANAGEMENT  ESTIMATE 
AT  COMPLETION  |1) 


CONTRACT  BUDGET 
BASEf, 


a.  NAME  (Last,  First,  Middle  Initial) 

Jim  Strait 

b.  TITLE 

US  LHC  Project  Manager 

c.  SIGNATURE 

d.  DATE  SIGNED 

(YYMMDD) 

990323 

1  8.  PERFORMANCE  DATA 

ITEM 

(D 

CURRENT  PERIOD 

CUMULATIVE  TO  DATE 

iREPROGRAMMINGj 

AT  COMPLETION 

BUDGET 

ED  COST 

ACTUAL 

COST  WORK 

PERFORMED 

(4) 

VAR1 

ANCE 

BUDGETED 

COST 

ACTUAL 

COST  WORK 

PERFORMED 

13) 

VARL 

ANCE 

ADJUS' 

rMENTS 

BUDGETED 

(14) 

ESTIMATED 

(15) 

VARIANCE 

(16) 

WORK 

SCHEDULED 

12) 

WORK 

PERFORMED 

(3) 

SCHEDULE 

(5) 

COST 

(6) 

WORK 

SCHEDULED 

m 

WORK 

PERFORMED 

18) 

SCHEDULE 

(10) 

COST 

(11) 

COST 

VARIANCE 

(12) 

BUDGET 

(13) 

a.  WORK  BREAKDOWN 

STRUCTURE  ELEMENT 

1.1-  Interaction  Regions 

2 

621.5 

307.8 

664  0 

-313.7 

-356.2 

11.283.0 

8.7020 

10.064.7 

-2.400. 1 

-1271.8 

35.019.7 

34.701.4 

1218.3 

1.1.1  -  Quadripoles 

3 

430.5 

228.5 

535.2 

-211.0 

-306.7 

7.083.0 

6.777.6 

7.761.3 

-1.205.4 

-0837 

21.004.7 

21.682.9 

221.8 

1.1. 1.1  -  Tooling 

4 

100.6 

21.3 

38.3 

-882 

-15.0 

1.070.8 

017.8 

1.018.4 

-153.1 

-100. 6 

1.606.4 

1.554.0 

52.4 

1.1. 1.2 -Cold  Mass 

4 

17.3 

21.8 

240.4 

4.5 

-218.6 

1.436.1 

1.380.4 

2431.0 

-55.8 

-1.050.6 

4.056.2 

5.051.1 

-904.9 

1.1. 1.3 -Cryostat 

4 

3.8 

0.0 

6.1 

-3.8 

-6.1 

123.8 

105.0 

488.7 

-17.9 

-380.0 

3298.4 

3.661.3 

-362.9 

1 . 1 . 1 .4  -  Magnet  Testing 

4 

154.2 

0.7 

15.8 

-153.5 

-15.1 

440.8 

215.0 

325.0 

-233.0 

-100.1 

2.675.8 

2.550.9 

124.8 

1.1.1.5-CaWe  &  Wedges 

4 

72 

0.5 

10.4 

-6.7 

-18.0 

1.260.0 

858.0 

710.4 

-401.1 

148.4 

1.483.7 

9342 

549.5 

1.1. 1.6-  Shipping 

4 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

380.9 

380.9 

0.0 

1.1. 1.7  -  EDIA 

4 

147.4 

184.1 

217.2 

36.7 

-33.1 

3.642.5 

3.298.8 

2.789.7 

-343.7 

509.1 

8.403.4 

7.550.6 

852.8 

1.1.2  -Dipoles 

3 

136.1 

600 

77.8 

- .  - 

-8.0 

2.203.0 

1.124.0 

1.425.5 

-1.079.1 

-300  6 

5.150.6 

4.372.2 

778.4 

1. 1.2.1  -Tooling 

4 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

18.6 

0.0 

-18.6 

158.8 

177.4 

-18.6 

1.1.2.2-01  Production 

4 

46.0 

15j0 

22.4 

-31.3 

-8.0 

375.6 

101.8 

324.3 

-183.8 

-132.5 

684.9 

633.6 

51.3 

1 . 1 .2.3  -  D2  Production 

4 

72.2 

54.2 

40.5 

-18.0 

4.7 

1.800.0 

844.0 

1.015.4 

-846.0 

-171.4 

2.453.9 

1.778.4 

675.5 

1 . 1 .2.4  -  Magnet  Testing 

4 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

5272 

527.2 

0.0 

1.1. 2.5 -EDIA 

4 

17.0 

0.0 

5.8 

-17.0 

-5.8 

137.5 

89.1 

672 

-48.4 

21.9 

1.325.9 

1255.6 

70.3 

CLASSIFICATION  (When  filled  in) 
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Appendix  B:  Level  3  EAC  Analysis 


AEHF  Level  3  $  EAC  Change  ($K) 


WBS  Activitv 

EAC  Change 

%  Total  EAC 

1.2:  EHF  PAYLOAD 

S97 1,461 

36% 

1.2A:  EHF  PAYLOAD 

S442,523 

7% 

2.1:  MISSION  OPERATIONS  ELEMENT 

$365,057 

,o 

© 

•  < 

7.3:  INTERIM  MAINTENANCE 

S276,573 

4% 

1.1:  SPACECRAFT  BUS 

S233,617 

7% 

AEHF  Level  3  %  EAC  Change 


WBS  Activity 

EAC  Change 

%  Total  EAC 

1.4:  SPACE  VEH  AGE  MAGE 

852% 

1% 

1.6:  SPACE  VEH  I&T 

690% 

2% 

1.5:  SPACE  VEH  SE  PM 

233% 

1% 

1.3:  LAUNCH  SUPPORT  OPERATIONS 

233% 

0.4% 

3.1:  SYSTEM  ENGINEERING  -  16 

160% 

3% 

FAB-T  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.01.01:  MODEM  PROCESSOR  GROUP  (MPG) 

$209,891 

31% 

1.01.03:  ANTENNA 

S129A36 

16% 

1.03.02:  PROGRAM  MANAGEMENT 

$79,225 

21% 

1.01.07:  INTEGRATION  ASSEMBLY  TEST  AND  CHECKOUT 

S75,871 

6% 

1.03.01:  SYSTEMS  ENGINEERING 

S46,646 

9% 

FAB-T  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.01.07:  INTEGRATION  ASSEMBLY  TEST  AND  CHECKOUT 

572% 

6% 

1.12.02:  MAJOR  SUBCONTRACTOR  UNDISTRIBUTED  BUDGET 

274% 

0.1% 

1.01.04:  ANCILLARY  EQUIPMENT 

227% 

1% 

1.04.01:  DEVELOPMENT  TEST  AND  EVALUATION 

148% 

3% 

1.01.03:  ANTENNA 

121% 

16% 
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GPS  III  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.2.3:  COMMUNICATION  /  PAYLOAD 

$295,865 

35% 

1.2.1:  SEIT  &  PM  &  OTHER  COMMON  ELEM 

$154,365 

26% 

1.2.2:  SPACECRAFT  BUS 

$149,873 

27% 

1.2.7:  LAUNCH  OPS  &  MISSION  SPT 

$53,721 

4% 

1.1.6:  PECULIAR  SUPPORT  EQUIPMENT 

$12,622 

2% 

GPS  III  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.2.7:  LAUNCH  OPS  &  MISSION  SPT 

333% 

4% 

1.1.4:  TRAINING 

181% 

0.1% 

1.2.3:  COMMUNICATION  /  PAYLOAD 

73% 

35% 

1.1.6:  PECULIAR  SUPPORT  EQUIPMENT 

52% 

2% 

1.2.1:  SEIT  &  PM  &  OTHER  COMMON  ELEM 

43% 

26% 

MUOS  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1500:  PAYLOAD  (PL) 

$429,616 

25% 

3100:  IG  SE  PMD 

$176,338 

12% 

4100:  SYSTEM  ENGINEERING,  INTEGRATION  &  TEST  (SEIT) 

$162,463 

11% 

1200:  SATELLITE  ASSEMBLY,  INTEGRATION  AND  TEST  (AI&T) 

$141,010 

6% 

3300:  GROUND  TRANSPORT  (GT) 

S120,163 

13% 

MUOS  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

5200:  PRE-DELIVERY  INTERIM  CONTRACTOR  SUPPORT 

328% 

2% 

1200:  SATELLITE  ASSEMBLY,  INTEGRATION  AND  TEST  (AI&T) 

264% 

6% 

3800:  USER  ENTRY  (UE) 

245% 

4% 

5100:  LOGISTICS  SUPPORT  AND  SUPPORTABILITY  DEV 

213% 

2% 

1100:  SATELLITE  SE  PM 

135% 

5% 

WGS  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1.1:  SPACECRAFT  BUS 

$226,239 

31% 

1.1.2:  COMMUNICATIONS  PAYLOAD 

$224,971 

37% 

1.2.2:  PROGRAM  MANAGEMENT 

$98,297 

16% 

1.1.4:  SATELLITE  VEH  INTEG  ASSY  AND  TEST  (CA) 

$73,704 

9% 

1.1.7:  LAUNCH  &  EARLY  ORBIT  OPERATIONS 

$42,251 

4% 
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WGS  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1.4:  SATELLITE  VEH INTEG  ASSY  AND  TEST  (CA) 

344% 

9% 

1.1.1:  SPACECRAFT  BUS 

281% 

31% 

1.2.2:  PROGRAM  MANAGEMENT 

153% 

16% 

1.1.2:  COMMUNICATIONS  PAYLOAD 

151% 

37% 

1.2.1:  SYSTEM  LEVEL  SYS  ENGRG 

42% 

1% 

NAVSTAR  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

131:  COMMON  HW 

S12,340 

15% 

152:  PROGRAM  MGMT 

$11,690 

15% 

121:  GB-GRAM-M  SSI  HARDWARE 

S8,214 

13% 

123:  GB-GRAM-M  SSI  IAT&C 

S4,922 

9% 

113:  GRAM  S  M  IAT&C 

$4,642 

7% 

NAVSTAR  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

131:  COMMON  HW 

274% 

15% 

152:  PROGRAM  MGMT 

194% 

15% 

113:  GRAM  S/M  IAT&C 

148% 

7% 

124:  GB-GRAM-M  SSI  SE/PM 

127% 

2% 

111:  GRAM  SM  HARDWARE 

127% 

3% 

GPS  OCX  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1  01.02:  PROGRAM  MANAGEMENT 

$22,954 

30% 

1.01.01:  SYSTEMS  ENGINEERING 

$10,689 

22% 

1.01.04:  SPECIAL  STUDIES 

$2,542 

1% 

2.02.01:  GPS  SYS/SYS  RQS  VERIF,  VALADATION 

$1,800 

1% 

1.18.03  DEVELOPMENT  LABS 

$1,256 

9% 

GPS  OCX  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

2.01.01:  SYSTEM  INTEGRATION  DEMOS 

890% 

0.4% 

2.03.01:  SYSTEM  OF  SYSTEMS  TEST  SUPPORT 

466% 

0.1% 

2.02.01:  GPS  SYS/SYS  RQS  VERIF,  VALADATION 

301% 

1% 

1.13.02:  MASTER  CONTROL  STATION  (MCS) 

90% 

1% 

1.18.07:  CREW  TRAINING  SYSTEM  (CTS) 

90% 

0.4% 
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WGS  C-10  Level  3  $  EAC  Change  ($K) 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1.2:  PROGRAM  MANAGEMENT 

S5.195 

31% 

1.2.3:  PAYLOAD 

S4:814 

52% 

1.2.1:  SPACE  VEHICLE  SEIT  AND  SUPPORT  EQUIPMENT 

-S851 

4% 

1.2.2:  BUS 

-S236 

12% 

1.1.1:  END  TO  END  SYSTEMS  ENGINEERING 

-S120 

1% 

WGS  C-10  Level  3  %  EAC  Change 


WBS  ACTIVITY 

EAC  Change 

%  Total  EAC 

1.1.2:  PROGRAM  MANAGEMENT 

12% 

31% 

1.2.3:  PAYLOAD 

6% 

52% 

1.2.2:  BUS 

-1% 

12% 

1.1.1:  END  TO  END  SYSTEMS  ENGINEERING 

-5% 

1% 

1.2.1:  SPACE  VEHICLE  SEIT  AND  SUPPORT  EQUIPMENT 

-13% 

4% 

65 


Appendix  C:  Mid- Acquisition  EAC  Cost  Growth  by  Completion  Percentage 


Mid-Acquisition  PMP  EAC  Cost  Growth  by  Completion  Percentage 


Mid-Acquisition  Total  EAC  Cost  Growth  by  Completion  Percentage 
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